Blog
0 min read

From zero to production: a comprehensive guide to managing Snowflake with Terraform

Written by
Greg Hinc

Manual clicks don’t scale. As Snowflake environments grow, managing them through the UI or ad-hoc scripts quickly leads to drift, blind spots, and compliance risks. What starts as a quick fix often becomes a challenge that slows delivery and exposes the business to security gaps.

Infrastructure as Code with Terraform solves these challenges by bringing software engineering discipline to Snowflake management. Using Terraform’s declarative language, engineers define the desired state of their Snowflake environment, track changes with version control, and apply them consistently across environments. Terraform communicates with Snowflake’s APIs through the official snowflakedb/snowflake provider, translating configuration into the SQL statements and API calls that keep your platform aligned and secure.

This guide provides a complete walkthrough of how to manage Snowflake with Terraform. From provisioning core objects like databases, warehouses, and schemas to building scalable role hierarchies and implementing advanced governance policies such as dynamic data masking.

Section 1: bootstrapping Terraform for secure Snowflake automation

The initial setup of the connection between Terraform and Snowflake is the most critical phase of the entire process. A secure and correctly configured foundation is paramount for reliable and safe automation. This section focuses on establishing this connection using production-oriented best practices, specifically tailored for non-interactive, automated workflows typical of CI/CD pipelines.

1.1 The principle of least privilege: the terraform service role

Terraform should not operate using a personal user account. Instead, a dedicated service user must be created specifically for Terraform automation. Before any Terraform code can be executed, a one-time manual bootstrapping process must be performed within the Snowflake UI or via SnowSQL. This involves using the ACCOUNTADMIN role to create the dedicated service user and a high-level role for Terraform's initial operations.

The following SQL statements will create a TERRAFORM_SVC service user and grant it the necessary

-- Use the highest-level role to create users and grant system roles
USE ROLE ACCOUNTADMIN;

-- Create a dedicated service user for Terraform
-- The RSA_PUBLIC_KEY will be set in the next step
CREATE USER TERRAFORM_SVC
	TYPE = SERVICE
  COMMENT = 'Service user for managing Snowflake infrastructure via Terraform.'
  RSA_PUBLIC_KEY = '<YOUR_PUBLIC_KEY_CONTENT_HERE>';

-- Grant the necessary system roles to the Terraform service user
GRANT ROLE SYSADMIN TO USER TERRAFORM_SVC;
GRANT ROLE SECURITYADMIN TO USER TERRAFORM_SVC;

Granting SYSADMIN and SECURITYADMIN to the service user is a necessary starting point for the infrastructure management. The SYSADMIN role holds the privileges required to create and manage account-level objects like databases and warehouses. The SECURITYADMIN role is required for managing security principals, including users, roles, and grants.

1.2 Authentication: the key to automation

The choice of authentication method is important. The Snowflake provider supports several authentication mechanisms, including basic password, OAuth, and key-pair authentication. For any automated workflow, especially within a CI/CD context, key-pair authentication is the industry-standard and recommended approach.

A CI/CD pipeline, such as one running in GitHub Actions, is a non-interactive environment. Basic password authentication is a significant security risk and not recommended. This leaves key-pair authentication as the only method that is both highly secure, as it avoids transmitting passwords, and fully automatable.

The following table provides a comparative overview of the primary authentication methods available in the Snowflake provider, reinforcing the recommendation for key-pair authentication in production automation scenarios.

Table 1: Snowflake provider authentication methods

Method Primary Use Case Security Profile CI/CD Suitability
Password Local development, quick tests Low. Exposes credentials in state or environment variables. Low. Requires secure secret management; often blocked by MFA.
OAuth User-delegated access for third-party applications High. Token-based, short-lived credentials. Medium. Complex to set up for non-interactive server-to-server flows.
Key-Pair Recommended for Automation. Service accounts, CI/CD pipelines. High. Asymmetric cryptography; no passwords transmitted. High. Designed for secure, non-interactive authentication.

To implement key-pair authentication, an RSA key pair must be generated. The following openssl commands will create a 2048-bit private key in the required PKCS#8 format and its corresponding public key:

Bash

# Navigate to a secure directory, such as ~/.ssh
cd ~/.ssh

# Generate an unencrypted 2048-bit RSA private key in PKCS#8 format
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out snowflake_terraform_key.p8 -nocrypt

# Extract the public key from the private key
openssl rsa -in snowflake_terraform_key.p8 -pubout -out snowflake_terraform_key.pub

After generating the keys, the content of the public key file (snowflake_terraform_key.pub), including the -----BEGIN PUBLIC KEY----- and -----END PUBLIC KEY----- headers, must be copied and pasted into the ALTER USER statement from the previous step to associate it with the TERRAFORM_SVC user. For enhanced security, the private key itself can be encrypted with a passphrase. The Snowflake provider supports this by using the private_key_passphrase argument in the provider configuration.

1.3 Provider configuration: connecting Terraform to Snowflake

With the service user created and the key-pair generated, the final step is to configure the Snowflake provider in the Terraform project. This is typically done in a providers.tf file.

The foundational configuration requires defining the snowflakedb/snowflake provider and setting the connection parameters.

terraform {
  required_providers {
    snowflake = {
      source  = "snowflakedb/snowflake"
      version = ">= 2.8.0" // Best practice: pin to a major version to avoid breaking changes
    }
  }
}

provider "snowflake" {
  organization_name = var.snowflake_org_name
  account_name      = var.snowflake_account_name
  user              = var.snowflake_user         // e.g., "TERRAFORM_SVC"
  role              = "SYSADMIN"                 // Default role for the provider's operations
  authenticator     = "SNOWFLAKE_JWT"
  private_key       = var.snowflake_private_key
}

It is critical that sensitive values, especially the private_key, are never hardcoded in configuration files. The recommended approach is to define them as input variables marked as sensitive = true and supply their values through secure mechanisms like environment variables (e.g., TF_VAR_snowflake_private_key) or integration with a secrets management tool like GitHub Secrets or AWS Secrets Manager.

A common source of initial connection failures is the incorrect identification of the organization_name and account_name. These values can be retrieved with certainty by executing the following SQL queries in the Snowflake UI: SELECT CURRENT_ORGANIZATION_NAME(); and SELECT CURRENT_ACCOUNT_NAME();. Providing these simple but effective commands can prevent significant user frustration.

For more mature IaC implementations that strictly adhere to the principle of least privilege, Terraform supports the use of aliased providers. This powerful pattern allows for the definition of multiple provider configurations within the same project, each assuming a different role. This mirrors Snowflake's own best practices, where object creation (SYSADMIN) is separated from security management (SECURITYADMIN).

The following example demonstrates how to configure aliased providers:

# Default provider uses SYSADMIN for object creation (e.g., databases, warehouses)
provider "snowflake" {
  alias             = "sysadmin"
  organization_name = var.snowflake_org_name
  account_name      = var.snowflake_account_name
  user              = var.snowflake_user
  private_key       = var.snowflake_private_key
  authenticator     = "SNOWFLAKE_JWT"
  role              = "SYSADMIN"
}

# Aliased provider for security-related objects (e.g., roles, users, grants)
provider "snowflake" {
  alias             = "securityadmin"
  organization_name = var.snowflake_org_name
  account_name      = var.snowflake_account_name
  user              = var.snowflake_user
  private_key       = var.snowflake_private_key
  authenticator     = "SNOWFLAKE_JWT"
  role              = "SECURITYADMIN"
}

When using aliased providers, individual resource blocks must explicitly specify which provider to use via the provider meta-argument (e.g., provider = snowflake.securityadmin). This ensures that each resource is created with the minimum necessary privileges, enforcing a robust security posture directly within the code.

Section 2: provisioning core Snowflake infrastructure

Once the secure connection is bootstrapped, Terraform can be used to define and manage the fundamental building blocks of the Snowflake environment. This section provides code examples for creating databases, virtual warehouses, and schemas - the foundational components for any data workload.

2.1 Laying the foundation: databases

The database is the top-level container for schemas and tables in Snowflake. The snowflake_database resource is used to provision and manage these containers.

The following HCL example creates a primary database for analytics workloads, demonstrating the use of the aliased sysadmin provider and an optional parameter for data retention.

‍resource "snowflake_database" "analytics_db" {
  provider = snowflake.sysadmin // Explicitly use the sysadmin provider for object creation

  name    = "ANALYTICS"
  comment = "Primary database for analytics workloads managed by Terraform."

  // Optional: Configure Time Travel data retention period.
  // This setting can have cost implications.
  data_retention_time_in_days = 30
}

A core strength of Terraform is its ability to manage dependencies implicitly through resource references. In this example, once the analytics_db resource is defined, other resources, such as schemas, can reference its attributes (e.g., snowflake_database.analytics_db.name).

2.2 Compute power: warehouses

Virtual warehouses are the compute engines in Snowflake, responsible for executing queries and data loading operations. FinOps makes a difference, especially once usage grows.The snowflake_warehouse resource provides comprehensive control over their configuration, enabling a balance between performance and cost.

This example defines a standard virtual warehouse for analytics and business intelligence tools, showcasing parameters for cost optimization and scalability.

resource "snowflake_warehouse" "analytics_wh" {
  provider = snowflake.sysadmin

  name    = "ANALYTICS_WH"
  comment = "Warehouse for the analytics team and BI tools."

  // Define the compute capacity of the warehouse.
  warehouse_size = "X-SMALL"

  // Cost-saving measures: suspend the warehouse when idle.
  auto_suspend = 60 // Suspend after 60 seconds of inactivity.
  auto_resume  = true

  // Optional: Configure for multi-cluster for higher concurrency.
  min_cluster_count = 1
  max_cluster_count = 4
  scaling_policy    = "ECONOMY" // Prioritize conserving credits over starting clusters quickly.
}

The parameters in this resource directly impact both performance and billing. warehouse_size determines the raw compute power and credit consumption per second. auto_suspend is a critical cost-control feature, ensuring that credits are not consumed when the warehouse is idle. For workloads with high concurrency needs, the min_cluster_count, max_cluster_count, and scaling_policy parameters allow the warehouse to dynamically scale out to handle query queues, and then scale back in to conserve resources. Managing these settings via Terraform ensures that cost and performance policies are consistently applied and version-controlled.

2.3 Organizing your data: schemas

Schemas are logical groupings of database objects like tables and views within a database. The snowflake_schema resource is used to create and manage these organizational units.

The following HCL creates a RAW schema within the ANALYTICS database defined earlier.

resource "snowflake_schema" "raw_data" {
  provider = snowflake.sysadmin

  // Create an explicit dependency on the database resource.
  database = snowflake_database.analytics_db.name

  name    = "RAW"
  comment = "Schema for raw, unprocessed data ingested from source systems."
}

It is important to note that when a new database is created in Snowflake, it automatically includes a default schema named PUBLIC. While this schema is created outside of Terraform's management, administrators should be aware of its existence. For environments that require strict access control, it is a common practice to immediately revoke all default privileges from the

PUBLIC schema to ensure it is not used inadvertently. Terraform can be used to manage this revocation if desired, but the schema itself will not be in the Terraform state unless explicitly imported.

Section 3: mastering access control with role hierarchies

Effective access control is a cornerstone of data governance and security. Snowflake's Role-Based Access Control (RBAC) model is exceptionally powerful, particularly its support for role hierarchies. Managing this model via Terraform provides an auditable, version-controlled, and scalable approach to permissions management. This section details how to construct a robust RBAC framework using a best-practice model of functional and access roles. At scale, keeping this clean is less about writing the first version and more about maintaining standards over time, which is why Platform Team as a Service often owns RBAC and grants as the platform grows.

3.1 The building blocks: creating account roles

The foundation of the RBAC model is the creation of roles. A recommended pattern is to create two distinct types of roles:

  • Functional roles: These roles represent a job function or a persona, such as DATA_ANALYST or DATA_ENGINEER. Users are granted these roles.
  • Access roles: These roles represent a specific set of privileges on a specific set of objects, such as SALES_DB_READ_ONLY or RAW_SCHEMA_WRITE. These roles are granted to functional roles, not directly to users.

This separation decouples users from direct permissions, making the system vastly more scalable and easier to manage. The snowflake_account_role resource is used to create both types of roles

// Define a functional role representing a user persona.
resource "snowflake_account_role" "data_analyst" {
  provider = snowflake.securityadmin // Use the securityadmin provider for role management

  name    = "DATA_ANALYST"
  comment = "Functional role for users performing data analysis and reporting."
}

// Define an access role representing a specific set of privileges.
resource "snowflake_account_role" "analytics_db_read_only" {
  provider = snowflake.securityadmin

  name    = "ANALYTICS_DB_READ_ONLY"
  comment = "Grants read-only access to all objects in the ANALYTICS database."
}

3.2 Constructing the hierarchy: granting roles to roles

The true power of Snowflake's RBAC model is realized by creating hierarchies of roles. By granting access roles to functional roles, a logical and maintainable privilege structure is formed. If a data analyst needs access to a new data source, the corresponding access role is granted to the DATA_ANALYST functional role once, rather than granting privileges to every individual analyst. This pattern is essential for managing permissions at scale.

The snowflake_grant_account_role resource is used to create these parent-child relationships between roles. It is important to use this resource, as the older snowflake_role_grants resource is deprecated.

The following example demonstrates how to grant the ANALYTICS_DB_READ_ONLY access role to the DATA_ANALYST functional role, and then nest the functional role under the system SYSADMIN role to complete the hierarchy.

// Grant the access role to the functional role.
// This gives all members of DATA_ANALYST the privileges of ANALYTICS_DB_READ_ONLY.
resource "snowflake_grant_account_role" "grant_read_access_to_analyst" {
  provider = snowflake.securityadmin

  role_name        = snowflake_account_role.analytics_db_read_only.name
  parent_role_name = snowflake_account_role.data_analyst.name
}

// Grant the functional role to SYSADMIN to create a clear role hierarchy.
// This allows system administrators to manage and assume the functional role.
resource "snowflake_grant_account_role" "grant_analyst_to_sysadmin" {
  provider = snowflake.securityadmin

  role_name        = snowflake_account_role.data_analyst.name
  parent_role_name = "SYSADMIN"
}

3.3 Assigning privileges to access roles

With the role structure in place, the final step is to grant specific object privileges to the access roles. The snowflake_grant_privileges_to_account_role resource is a consolidated and powerful tool for this purpose. This resource has evolved significantly in the Snowflake provider; older versions required separate grant resources for each object type (e.g., snowflake_database_grant), which resulted in verbose and repetitive code. The modern resource uses a more complex but flexible block structure (on_account_object, on_schema, etc.) to assign privileges. Users migrating from older provider versions may find this a significant but worthwhile refactoring effort.

This example grants the necessary USAGE and SELECT privileges to the ANALYTICS_DB_READ_ONLY access role.

// Grant USAGE privilege on the database to the access role.
resource "snowflake_grant_privileges_to_account_role" "grant_db_usage" {
  provider          = snowflake.securityadmin
  account_role_name = snowflake_account_role.analytics_db_read_only.name
  privileges        = ["USAGE"]
  
  on_account_object {
    object_type = "DATABASE"
    object_name = snowflake_database.analytics_db.name
  }
}

// Grant USAGE privilege on the schema to the access role.
resource "snowflake_grant_privileges_to_account_role" "grant_schema_usage" {
  provider          = snowflake.securityadmin
  account_role_name = snowflake_account_role.analytics_db_read_only.name
  privileges        = ["USAGE"]

  on_schema {
    // Use the fully_qualified_name for schema-level objects.
    schema_name = snowflake_schema.raw_data.fully_qualified_name
  }
}

// Grant SELECT on all existing tables in the schema.
resource "snowflake_grant_privileges_to_account_role" "grant_all_tables_select" {
    provider          = snowflake.securityadmin
    account_role_name = snowflake_account_role.analytics_db_read_only.name
    privileges        = ["SELECT"]
    
    on_schema_object {
        all {
            object_type_plural = "TABLES"
            in_schema          = snowflake_schema.raw_data.fully_qualified_name
    }
  }
}

// Grant SELECT on all FUTURE tables created in the schema.
resource "snowflake_grant_privileges_to_account_role" "grant_future_tables_select" {
  provider          = snowflake.securityadmin
  account_role_name = snowflake_account_role.analytics_db_read_only.name
  privileges        = ["SELECT"]

  on_schema_object {
    future {
      object_type_plural = "TABLES"
      in_schema          = snowflake_schema.raw_data.fully_qualified_name
    }
  }
}

A particularly powerful feature demonstrated here is the use of the future block. Granting privileges on future objects ensures that the access role will automatically have the specified permissions on any new tables created within that schema. This dramatically reduces operational overhead, as permissions do not need to be manually updated every time a new table is deployed. However, it is important to understand Snowflake's grant precedence: future grants defined at the schema level will always take precedence over those defined at the database level. This can lead to "insufficient privilege" errors if not managed carefully across different roles and grant levels.

3.4 An optional "Audit" role for bypassing data masks

In certain scenarios, such as internal security audits or compliance reviews, it may be necessary for specific, highly-trusted users to view data that is normally protected by masking policies. Creating a dedicated "audit" role for this purpose provides a controlled and auditable mechanism to bypass data masking when required.

This role should be considered a highly privileged functional role and granted to users with extreme care.

// Define a special functional role for auditing PII data.
resource "snowflake_account_role" "pii_auditor" {
  provider = snowflake.securityadmin

  name    = "PII_AUDITOR"
  comment = "Functional role for users who need to view unmasked PII for audit purposes."
}

Crucially, creating this role is not enough. For it to be effective, every relevant masking policy must be explicitly updated to include logic that unmasks data for members of the PII_AUDITOR role. This ensures that the ability to view sensitive data is granted on a policy-by-policy basis. An example of how to modify a masking policy to incorporate this audit role is shown in the following section.

Section 4: advanced data governance with dynamic data masking

Moving beyond infrastructure provisioning, Terraform can also codify and enforce sophisticated data governance policies. Snowflake's Dynamic Data Masking is a powerful feature for protecting sensitive data at query time. By managing these policies with Terraform, organizations can ensure that data protection rules are version-controlled, auditable, and consistently applied across all environments.

4.1 Defining the masking logic

A masking policy is a schema-level object containing SQL logic that determines whether a user sees the original data in a column or a masked version. The decision is made dynamically at query time based on the user's context, most commonly their active role.

The snowflake_masking_policy resource is used to define this logic. The policy's body contains a CASE statement that evaluates the user's session context and returns the appropriate value.

The following example creates a policy to mask email addresses for any user who is not in the DATA_ANALYST or PII_AUDITOR role.

resource "snowflake_masking_policy" "email_mask" {
  provider = snowflake.sysadmin // Policy creation often requires SYSADMIN or a dedicated governance role

  name     = "EMAIL_MASK"
  database = snowflake_database.analytics_db.name
  schema   = snowflake_schema.raw_data.name
  
  // Defines the signature of the column the policy can be applied to.
  // The first argument is always the column value to be masked.
  argument {
    name = "email_val"
    type = "VARCHAR"
  }
  
  // The return data type must match the input data type.
  return_type = "VARCHAR"

  // The core masking logic is a SQL expression.
  body = <<-EOF
    CASE
      WHEN IS_ROLE_IN_SESSION('DATA_ANALYST') OR IS_ROLE_IN_SESSION('PII_AUDITOR') THEN email_val
      ELSE '*********'
    END
  EOF

  comment = "Masks email addresses for all roles except DATA_ANALYST and PII_AUDITOR."
}

The SQL expression within the body argument offers immense flexibility. It can use various context functions (like CURRENT_ROLE() or IS_ROLE_IN_SESSION()) and even call User-Defined Functions (UDFs) to implement complex logic. However, this flexibility means the logic itself is not validated by Terraform's syntax checker; it is sent directly to Snowflake for validation during the

terraform apply step. It is also a strict requirement that the data type defined in the argument block and the return_type must match the data type of the column to which the policy will eventually be applied.

4.2 Applying the policy to a column

Creating a masking policy is only the first step; it does not protect any data on its own. The policy must be explicitly applied to one or more table columns. This crucial second step is often a point of confusion for new users, who may create a policy and wonder why data is still unmasked. The snowflake_table_column_masking_policy_application resource creates this essential link between the policy and the column.

The following example demonstrates how to apply the EMAIL_MASK policy to the EMAIL column of a CUSTOMERS table.

// For this example, we assume a 'CUSTOMERS' table with an 'EMAIL' column
// already exists in the 'RAW' schema. In a real-world scenario, this table
// might also be managed by Terraform or by a separate data loading process.
// We use a data source to reference this existing table.
data "snowflake_table" "customers" {
  database = snowflake_database.analytics_db.name
  schema   = snowflake_schema.raw_data.name
  name     = "CUSTOMERS"
}

// Apply the masking policy to the specific column.
resource "snowflake_table_column_masking_policy_application" "apply_email_mask" {
  provider = snowflake.sysadmin

  table_name  = "\"${data.snowflake_table.customers.database}\".\"${data.snowflake_table.customers.schema}\".\"${data.snowflake_table.customers.name}\""
  column_name = "EMAIL" // The name of the column to be masked

  masking_policy_name = snowflake_masking_policy.email_mask.fully_qualified_name
  
  // An explicit depends_on block ensures that Terraform creates the policy
  // before attempting to apply it, preventing race conditions.
  depends_on = [
    snowflake_masking_policy.email_mask
  ]
}

This two-step process—defining the policy logic and then applying it - provides a clear and modular approach to data governance. The same policy can be defined once and applied to many different columns across multiple tables, ensuring that the masking logic is consistent and centrally managed.

Conclusion: the path to mature Snowflake IaC

This guide has charted a course from the initial, manual bootstrapping of a secure connection to the automated provisioning and governance of a production-grade Snowflake environment. To ensure the long-term success and scalability of managing Snowflake with Terraform, several key practices should be adopted as standard procedure:

  • Version control: All Terraform configuration files must be stored in a version control system like Git. This provides a complete, auditable history of all infrastructure changes and enables collaborative workflows such as pull requests for peer review before any changes are applied to production.
  • Remote state management: The default behaviour of Terraform is to store its state file locally. In any team or automated environment, this is untenable. A remote backend, such as an Amazon S3 bucket with a DynamoDB table for state locking, must be configured. This secures the state file, prevents concurrent modifications from corrupting the state, and allows CI/CD pipelines and team members to work from a consistent view of the infrastructure.
  • Modularity: As the number of managed resources grows, monolithic Terraform configurations become difficult to maintain. Code should be refactored into reusable modules. For instance, a module could be created to provision a new database along with a standard set of access roles and default schemas. This promotes code reuse, reduces duplication, and allows for more organized and scalable management of the environment.
  • Provider versioning: The Snowflake Terraform provider is actively evolving. To prevent unexpected breaking changes from new releases, it is crucial to pin the provider to a specific major version in the terraform block (e.g., version = "~> 2.8"). This allows for intentional, planned upgrades. When upgrading between major versions, it is essential to carefully review the official migration guides, as significant changes, particularly to grant resources, may require a concerted migration effort.

With this robust foundation in place, the path is clear for expanding automation to encompass even more of Snowflake's capabilities. The next logical steps include using Terraform to manage snowflake_network_policy for network security, snowflake_row_access_policy for fine-grained data filtering, and snowflake_task for orchestrating SQL workloads. Ultimately, the entire workflow should be integrated into a CI/CD pipeline, enabling a true GitOps model where every change to the Snowflake environment is proposed, reviewed, and deployed through a fully automated and audited process. By embracing this comprehensive approach, organizations can unlock the full potential of their data platform, confident in its security, scalability, and operational excellence.

Why Snowstack for Terraform and Snowflake

Automation without expertise can still fail. Terraform gives you the tools, but it takes experience and the right design patterns to turn Snowflake into a secure, cost-efficient, and scalable platform. The hard part is deciding what “good” looks like in your Snowflake account and making that repeatable across teams, environments, and change cycles.

That is where Snowstack comes in. As a Snowflake-first consulting partner, we help organizations move beyond trial-and-error scripts to fully automated, production-grade environments. Our engineers design secure architectures, embed Terraform best practices, and ensure governance and cost controls are built in from day one.

FAQs

Start with the stable building blocks: warehouses, databases, schemas, roles, and grants. Codifying these gives you a repeatable baseline, clearer reviews, and fewer “who changed what” moments. If you want help turning that into a production standard, our Snowflake consulting is built for exactly that.

Yes. A dedicated service user keeps changes auditable, avoids personal-account dependencies, and makes it easier to enforce least privilege. We usually set this up during a Snowflake implementation so the roles and permissions are clean from the start.

Key pair authentication is the go-to approach for automation because it avoids passwords and supports key rotation. The real “gotchas” are how you store the private key, how you rotate it without downtime, and how you lock down the service role. If you want a proven baseline, that’s a common starting point in Snowflake consulting.

Drift usually comes from “out-of-band” changes in Snowsight, differences in naming/identifiers, or having more than one resource trying to own the same privilege set. The fix is to pick one source of truth, standardize object identifiers, and keep grants/ownership patterns consistent. When teams need ongoing ownership, that’s what Platform Team as a Service is for.

Make cost controls part of the code, not tribal knowledge. That means codifying warehouse sizing rules, auto suspend, scaling settings, and environment defaults, then pairing it with a usage review cadence. If savings is the goal, this is exactly the focus of our FinOps work.

Provider upgrades can include grant model changes and deprecations, which can break older configurations if you’re still using removed resources. Treat upgrades like a controlled migration: update in steps, migrate grant resources intentionally, and avoid mixing old and new grant patterns in the same scope. If you want us to review the upgrade path before you roll it out, start with Snowflake consulting.

Yes, but do it in phases. First codify a baseline (core warehouses, core roles, and standard grants), then gradually bring the rest under management through imports or controlled rebuilds. The goal is to reduce risk, not “flip a switch” overnight. We typically approach this through Snowflake consulting or a scoped implementation.

Learn more about Snowflake from top experts

Join data leaders who get Snowflake insights and updates delivered straight to their inbox.

Thanks for joining us!

We’ll keep you posted with fresh updates and resources.

Oops! Something went wrong while submitting the form.
Insights

Learnings for data leaders

Blog
5 min read

How Snowflake is different from other databases: 3 architecture advantages for modern data teams

Some companies still run databases like it’s 1999. Others have adopted cloud-native architectures that cut costs in half and double performance. Guess who’s winning?

Read more

Some companies still run databases like it’s 1999. Others have adopted cloud-native architectures that cut costs in half and double performance. Guess who’s winning?

Traditional databases force a trade-off between performance and budget. Collaboration still means passing around CSVs. Forward-thinking organizations have shifted to Snowflake’s cloud-native architecture, which scales instantly, operates securely, and keeps costs under control. But what truly sets Snowflake apart from traditional databases or even other cloud data platforms?

In this blog, we’ll break down three key architectural advantages that make Snowflake a game-changer for businesses that want to migrate to the cloud.

But first, what is a cloud-native database?

A cloud-native database is designed from the ground up for the cloud. Unlike traditional databases that were adapted from on-premise systems, cloud-native platforms are purpose-built to take advantage of the cloud’s strengths: scalability, flexibility, and resilience.

They scale horizontally by adding capacity in parallel instead of relying on bigger machines. They automatically adjust resources up or down based on demand, so you only pay for what you use. They also come with built-in high availability through data replication and automated recovery.

In short, a cloud-native database removes the rigid trade-offs of legacy systems and gives modern businesses the performance, efficiency, and reliability they need to stay competitive.

Snowflake's architecture: 3 strategic advantages

Snowflake isn’t just faster or cheaper. It’s built differently. The three architectural choices below explain why modern data teams trust Snowflake to scale, collaborate, and deliver insights in ways legacy systems never could.

1. Separation of storage and compute: elasticity without trade-offs

Most databases tie storage and compute together. Need more power to run quarterly reports? You’ll also pay for storage you don’t use. Want to keep historical data at a lower cost? You’re still paying for compute you don’t actually need.

Snowflake's Solution: Snowflake's architecture fundamentally decouples storage and compute layers, creating unprecedented flexibility for modern data teams.

  • You can scale compute resources up or down independently of your data storage.
  • Multiple workloads (e.g., data ingestion, analytics queries, and reporting) can run simultaneously on isolated compute clusters without performance conflicts.
  • You can assign different warehouses (compute clusters) to different teams or departments without worrying about concurrency issues or resource contention.

Business impact: Imagine a BI team that runs heavy dashboards while a data science team trains models on the same data. The beauty behind this separation is that both can operate without stepping on each other’s toes. This translates to faster time-to-insight, cost control, and happy teams who aren’t waiting for resources to free up.

2. Multi-cluster shared data architecture: built for collaboration and scale

Traditional databases become performance challenge as more users access the system. Query response times degrade, teams queue for resources, and data silos emerge as different departments seek workarounds.

Snowflake's Solution: Snowflake’s multi-cluster shared data model allows any number of users and tools to access the same single source of truth without performance degradation. The platform automatically manages concurrency through intelligent multi-cluster compute scaling.

What this means for data teams:

  • Unlimited concurrency: Teams don’t have to wait in line to access the warehouse. Snowflake automatically adds compute clusters as needed and scales them back down when demand drops.
  • Cross-team collaboration: Data Engineers, analysts, and ML engineers can work off the same dataset in real time, using SQL, Python, or third-party tools.
  • Data sharing across organizations: Snowflake’s architecture supports secure data sharing with external partners or vendors without copying or moving data. You simply grant access.

Business impact: This makes Snowflake not just a warehouse but a collaboration platform for data. Whether your team is distributed across continents or collaborating with external partners, Snowflake enables fast, consistent, and secure access to data.

3. Zero management with cloud-native infrastructure

Managing a traditional database means dealing with provisioning, tuning, indexing, patching, and more. These tasks require specialized DBAs and often lead to downtime, delays, and human error.

Snowflake flips the script with a “zero-management” approach.

Thanks to its fully managed SaaS model:

  • No infrastructure to manage. Snowflake runs entirely in the cloud (on AWS, Azure, or GCP), abstracting away the underlying hardware.
  • Automatic tuning and optimization. No need to manually set indexes or optimize queries, Snowflake handles that under the hood.
  • Security and compliance out of the box. Features like automatic encryption, role-based access control, and compliance with standards (HIPAA, GDPR, SOC 2) are built-in.

Business impact: This lets your team focus on data and insights, not on maintenance. IT teams no longer need to waste time on low-value operational tasks. Instead, they can accelerate innovation and reduce costs.

Snowflake vs. the competition: why architecture matters

In 2025, your data architecture is more than a technical choice. It is a strategic decision that defines how quickly your organization can compete, innovate, and scale. When you compare modern data platforms, Snowflake's architectural advantages become clear when compared to alternatives:

How Snowflake’s architecture drives results?

Snowflake’s architecture solves the trade-offs that hold traditional databases back and delivers flexibility that many cloud platforms still lack. But technology alone is not enough. The difference comes from how you implement it.

Take the case of a $200M pharmaceutical distributor. Their teams were stuck with siloed on-prem systems, compliance risks, and reports that took hours to run. Our Snowflake-certified experts helped them migrate to Snowflake’s cloud-native architecture with a single governed data layer, dedicated compute clusters, and built-in role-based access. In just 90 days, reporting was 80% faster, the architecture was ready for AI and advanced analytics, and teams finally worked from the same source of truth.

👉 Read the full case study here

Making Snowflake’s architecture work for your business

Every organization’s data challenges look different, but the goal is the same: to turn Snowflake into a platform that delivers measurable results. That’s where Snowstack comes in. We bring proven experience from complex projects in finance, pharma, and FMCG. This gives clients confidence that their architecture is designed for scale, collaboration, and compliance from day one. Our role goes beyond implementation. We act as a long-term partner who helps data teams adapt, optimize, and grow with Snowflake as business needs evolve.

Are you getting the full value from Snowflake’s architecture?

FAQs

Snowflake's architecture separates storage and compute into independent layers, unlike traditional databases that tightly couple these resources. This means you can scale processing power without paying for additional storage, and store massive amounts of data without impacting query performance. Snowflake also provides unlimited concurrency through multi-cluster compute, automatic optimization, and zero infrastructure management.

Snowflake focuses on data warehousing, BI, and analytics with SQL-first approach and zero management overhead. Databricks specializes in data science, machine learning, and complex analytics with notebook-based development. Check our blog to explore the differences.

Snowflake uses a consumption-based pricing model with separate charges for storage and compute . You pay for data storage based on the amount stored (compressed), and compute costs based on the size and duration of warehouse usage. Credits are consumed only when warehouses are actively running queries. Check our blog to find out how you can optimize your data warehouse costs.

No, Snowflake is a cloud-native platform that runs exclusively on AWS, Azure, and Google Cloud Platform. However, this cloud-only approach is actually an advantage. It eliminates the infrastructure management overhead, provides automatic scaling, and ensures you always have access to the latest features and security updates without manual maintenance.

Yes, Snowflake natively supports semi-structured and unstructured data formats including JSON, XML, Parquet, Avro, and even binary data like images and documents.

Implementation timelines vary based on data complexity and organizational requirements. Simple migrations can be completed in 4–8 weeks, while comprehensive enterprise transformations typically take 3–6 months. Using proven frameworks and experienced implementation partners like Snowstack can significantly accelerate timelines while reducing risks and ensuring best practices from the start.

Snowflake runs natively on AWS, Azure, and Google Cloud Platform, using each cloud provider's infrastructure while maintaining a consistent experience across all platforms. You can even replicate data across different cloud regions or providers for disaster recovery and compliance requirements. Snowflake handles all the underlying infrastructure complexity, so you focus on your data, not cloud management.

Yes, Snowflake integrates with SQL Server, Oracle, and virtually any database through various methods: direct connectors, ETL tools like Fivetran or Informatica, custom APIs, and batch file transfers. Many organizations use Snowflake as their central data warehouse while keeping operational systems on SQL Server or Oracle, replicating data through automated pipelines.

Blog
5 min read

Why your Snowflake agents give wrong answers on good data

Your Snowflake agent gives wrong answers on clean data because it knows your schema, not your business - here's the context layer that fixes it.

Read more

Under the Hood: grounding CoWork with Cortex Sense — not just prompting it

Your agent isn't wrong because your data is dirty. It's wrong because it doesn't know what your data means. An LLM can read your schema perfectly — table names, column types, row counts — and still have no idea that net revenue means gross revenue after discounts, that the fiscal year starts in February, or that "active customer" excludes anyone who churned last quarter. That gap between the schema an agent sees and the business meaning it doesn't is where confident, wrong answers come from. Closing it is now the single highest-leverage thing a data team can do for AI.

That's also the thesis Snowflake built its entire Summit 2026 agentic story around.

What actually changed at Summit 2026

Two things matter for anyone running agents on Snowflake:

Snowflake Intelligence is now CoWork. Same product lineage — the personal work agent that decomposes a question, researches across structured and unstructured data, and returns a cited answer — new name. If you saw Episode 3, this is the thing you already built against. Existing deployments migrate automatically.

Cortex Sense is the headline, and it's about accuracy, not features. Cortex Sense is a runtime context-enrichment layer: it automatically assembles business context — query history, object metadata, BI dashboards, and Horizon Context semantic views — and feeds it to CoWork and CoCo at query time, with no manual configuration. Snowflake's own internal benchmark puts the difference starkly: 47% accuracy on complex enterprise queries without it, 83% with it— and just 23% for frontier coding agents wired up through Snowflake's MCP connector alone. The message Snowflake is sending could not be clearer: context, not the model, determines agent quality.

We agree with that framing. But there are two catches, and they're exactly where a data team's real work lives.

The two catches nobody puts on the keynote slide

Catch #1 — Cortex Sense is private preview (as of June 2026). CoWork is shipping to enterprises now; Cortex Sense is not generally available yet. So the default CoWork deployment today operates closer to that 47% baseline, withoutSnowflake's own context infrastructure at production readiness. You can't wait for the feature to flip on and rescue answer quality before your stakeholders start trusting (or distrusting) the agent.

Catch #2 — even at GA, Cortex Sense is only as good as what's underneath it. Read the description again: it assembles context from your semantic views, metadata, and dashboards. If those definitions are missing, ambiguous, or contradictory, Cortex Sense faithfully assembles ambiguous context. And there's a deeper trap that governance alone never solves: access control is not correctness. RBAC enforces who can query the revenue table; it says nothing about whether that table is accurate, consistently defined, or current. An agent querying a revenue figure with an upstream ingestion error will return a confident, beautifully-cited, wrong number — and every guardrail will have done its job.

So the work is the same whether Cortex Sense is in preview or GA: you build the governed context layer and you make sure the data beneath it is actually right. The good news is that this work is not throwaway — the semantic layer you build now is precisely the substrate Cortex Sense consumes later. You're not waiting for the feature; you're getting ahead of it.

Here's how we build it.

Under the Hood: the context layer, step by step

Step 1 — Put the business definitions in a governed semantic view

A semantic view is a schema-level Snowflake object that maps physical columns to business concepts — facts, dimensions, and metrics — and stores the definitions natively, under RBAC, where both Cortex Analyst and (eventually) Cortex Sense read them. This is where you kill ambiguity once, centrally, instead of in fifty different dashboards.

The canonical example is the one Snowflake itself uses: revenue is physically stored in a column called amt_ttl_pre_dsc, but the business always means gross revenue after discounts. You encode that once:

CREATE OR REPLACE SEMANTIC VIEW analytics.sales.revenue_model
  TABLES (
    orders AS prod.sales.orders
      PRIMARY KEY (order_id)
      WITH SYNONYMS ('sales', 'bookings')
      COMMENT = 'One row per order line. Source of truth for revenue.',
    unit AS prod.sales.business_unit_dim
      PRIMARY KEY (unit_id)
  )
  RELATIONSHIPS (
    orders_to_unit AS orders (unit_id) REFERENCES unit (unit_id)
  )
  FACTS (
    orders.gross_amount   AS amt_ttl_pre_dsc,
    orders.discount_rate  AS disc_rate
  )
  DIMENSIONS (
    unit.unit_name    AS unit_name WITH SYNONYMS ('business unit', 'segment'),
    orders.order_date AS order_dt
  )
  METRICS (
    orders.net_revenue AS SUM(orders.gross_amount * (1 - orders.discount_rate))
      COMMENT = 'Net revenue = gross revenue after discounts. Use this for ALL
                 revenue reporting. Never sum amt_ttl_pre_dsc directly.'
  )
  COMMENT = 'Governed revenue model. These definitions are the single source
             of truth for agents and BI alike.';

Now anyone — human or agent — asks the question the same way and gets the same number:

SELECT * FROM SEMANTIC_VIEW (
  analytics.sales.revenue_model
  METRICS    net_revenue
  DIMENSIONS unit_name
);

Step 2 — Write your comments like prompts, because they are

This is the part that separates "it compiles" from "the agent is actually right." In a semantic view, Cortex Analyst reads your COMMENT text as instructions, not documentation. The comment on net_revenue above isn't a note for a future engineer — it's telling the model which column is not revenue. Be that explicit everywhere: define what a metric means, when to use it, and what to avoid. If you don't write it down, the model guesses, and a guess is how you get a wrong answer on clean data.

Two more high-leverage moves on the same object:

  • Synonyms so "business unit," "segment," and "BU" all resolve to one dimension. Agents fail constantly on vocabulary mismatch; this fixes it cheaply.
  • Verified queries — known-good question/SQL pairs that anchor the model on your hardest or most political metrics:

-- inside CREATE SEMANTIC VIEW, after the COMMENT clause:
AI_VERIFIED_QUERIES (
  net_rev_by_unit AS (
    QUESTION  'What was net revenue by business unit last quarter?'
    VERIFIED_AT 1717200000
    VERIFIED_BY '(owner = data-platform@yourco.com)'
    SQL 'SELECT * FROM SEMANTIC_VIEW (analytics.sales.revenue_model
           METRICS net_revenue DIMENSIONS unit_name)'
  )
);

One discipline worth stating plainly: only add verified queries you have actually validated. One wrong example teaches the model a bad habit at scale.

Step 3 — Measure the lift on your KPIs, don't take 47→83 on faith

Snowflake's benchmark is theirs, on their data. Before you tell your CFO the agent is trustworthy, prove it on your questions. Snowflake ships a Cortex Agent evaluation framework for exactly this — define a dataset of real questions with expected answers, then score the agent against it:

evaluation:
  agent_params:
    agent_name: "revenue_agent"
    agent_type: "CORTEX AGENT"
  run_params:
    label: "Baseline — before semantic layer"
    source_metadata:
      type: "dataset"
      dataset_name: "kpi_eval_set"
  metrics:
    - answer_correctness        # how close the answer is to ground truth
    - tool_selection_accuracy   # did it call the right tools? (public preview)
    - logical_consistency       # reference-free; consistency across the run

Run it once before the semantic layer exists, run it again after. The delta is your evidence — and your regression test. Wire it into CI so a careless change to a metric definition can't silently re-break answer quality next month.

Step 4 — Fix the data the context layer points at

A perfect semantic layer over a stale or half-loaded table still produces a wrong answer, just a well-defined one. So the context work has a twin: source-to-report reconciliation, freshness checks, and catching the broken or partial feeds that quietly poison a metric. That's a whole topic — it's Episode 5 — but flag it now, because "the agent gave the wrong number" is at least as often an ingestion problem as a semantics problem.

What this means, by role

If you lead data or analytics: the semantic layer is no longer a BI nicety — it's the accuracy substrate for every agent you're about to be asked to deploy. Building it now pays twice: better Cortex Analyst answers today, and a ready-made context source for Cortex Sense when it GAs.

If you're the architect or lead engineer: treat semantic views as strict contracts, not flexible SQL. Model relationships explicitly, comment like you're prompting, anchor hard metrics with verified queries, and put an evaluation set in CI. This is the build work that makes the demo survive contact with production.

If you own the platform strategy (VP / CDO): the question your stakeholders are really asking is "can we trust this for a real decision?" The honest answer is "only as far as our governed definitions and our data quality go." That's a roadmap, not a blocker — and it's a far better place to invest than another model evaluation.

How we'd approach it

Most teams we talk to don't have a context problem they can see — they have a trust problem they can feel: two dashboards disagree, an agent answer doesn't match the board deck, nobody's quite sure which number is right. The fix starts with finding where the definitions diverge and where the data underneath is wrong, before pointing any agent at it.

That's the shape of our AI-readiness assessment — a fixed-scope first step that maps your sources, definitions, and the gaps between what your reports say and what your data actually contains, so the agents you ship are accurate by construction. If your CoWork answers are landing in the "confident but wrong" zone, that's the place to start.

FAQs

Because they understand the schema, not the business meaning. Without governed definitions for metrics, fiscal calendars, and segment rules, the model infers them — and inference is where confident, wrong answers originate. Take a look at our AI and Governance page for more details.

A runtime context-enrichment layer announced at Summit 2026 (June 2) that automatically assembles business context — query history, metadata, BI dashboards, and semantic views — and supplies it to CoWork and CoCo at query time. Snowflake's internal benchmark reports it lifts accuracy from 47% to 83% on complex enterprise queries. It is in private preview as of June 2026.

No. Cortex Sense draws on your semantic views and metadata. Building a governed semantic layer now improves Cortex Analyst answers today and becomes the exact substrate Cortex Sense consumes when it reaches GA.

No. RBAC and governance control who can access data; they do not certify that the data is correct, consistently defined, or current. An agent can be fully governed and still return a wrong number from a table with an upstream error.

For new work, semantic views — they're native schema-level objects with full RBAC, sharing, and catalog support. Legacy YAML semantic models still work with Cortex Analyst for backward compatibility.

Notes & sources: Cortex Sense status and the 47%→83% figure are from Snowflake's own materials and product announcements (Snowflake Summit 2026, June 2); Cortex Sense is in private preview as of June 2026, so treat the figure as a vendor benchmark and validate on your own data. Semantic-view DDL and the comment-as-instruction behavior follow Snowflake's CREATE SEMANTIC VIEW and semantic-view documentation; semantic-view SQL is stricter than ordinary SQL, so validate any DDL against current docs for your account version. Cortex Agent evaluation metrics (answer correctness, tool-selection accuracy, logical consistency) are from Snowflake's Cortex Agent evaluations documentation.

Blog
5 min read

Can Snowflake store unstructured data? How Snowflake handles documents, images, and other data in 2025

Snowflake isn’t just rows and columns anymore. In 2025 you can land PDFs, images, logs, and app data next to your tables, then query, enrich, and search them with SQL, Snowpark, and Cortex AI.

Read more

What if your PDFs, transcripts, and logs could live in the same place as your BI dashboards? For years, Snowflake was known primarily as a cloud native data warehouse built for structured analytics. It was the go-to solution for SQL analysts, BI teams, and data engineers working with neat rows and columns. Meanwhile, many teams dealing with documents, images, logs, and raw application data assumed they needed separate storage such as Amazon S3, Google Cloud Storage, Azure Blob, or NoSQL databases.

In 2025, that separation no longer has to exist. Snowflake is now a multimodal data platform that can store, process and query unstructured data.

So yes, Snowflake can store unstructured data, but more importantly, it can use it. This capability offers significant architectural advantages for modern data teams. In this blog post, we’ll break down exactly how and why it matters.

What is unstructured data?

Unstructured data refers to any information that doesn't fit neatly into traditional rows and columns. This includes:

  • Documents: PDF, DOCX, TXT files
  • Images: PNG, JPG, TIFF formats
  • Audio and video files: Media content and recordings
  • Logs and event data: Application and system logs
  • Communication data: Email threads and chat transcripts
  • Markup and structured text: HTML, XML, JSON blobs
  • Binary files: Application-specific file formats

As organisations increasingly generate massive volumes of this data, the need for unified platforms that can both store and analyse unstructured content has become critical.

How Snowflake stores unstructured data?

Snowflake stages for unstructured data

Snowflake manages unstructured data through stages. This means through storage locations that reference files either within Snowflake's managed infrastructure or in external cloud storage:

  • Internal Stages: Files are stored within Snowflake's managed storage, offering quick setup and seamless integration
  • External Stages: Files remain in external cloud locations (Amazon S3, Azure Blob Storage, Google Cloud Storage), with Snowflake accessing them via metadata references

You can also combine both approaches for optimal performance and scalability based on your specific requirements.

The FILE data type in Snowflake for unstructured files and metadata

Snowflake provides a dedicated FILE data type for unstructured data. A FILE value represents a reference to a file stored in an internal or external stage, without storing the actual file content in the table itself. This approach allows:

  • Efficient storage and cost management
  • Fast metadata querying
  • Seamless integration with processing pipelines

Accessing unstructured files in Snowflake

Snowflake provides familiar commands for file management:

  • PUT: Upload files to stages
  • GET: Download files from stages
  • LIST: View files stored in stages

These operations mirror cloud storage interactions while maintaining Snowflake's security and governance standards.

Processing and querying unstructured data in Snowflake

Storage is just the beginning. Snowflake's real power lies in its ability to process and extract insights from unstructured data.

Snowflake Cortex AI and Document AI for PDFs, images and hybrid search

Cortex AI enables advanced analytics on unstructured data directly within Snowflake:

  • Document analysis: Extract text, summarise content, and perform batch LLM inference on PDFs and documents
  • Image processing: Run classification and analysis on stored images
  • Multimodal SQL functions: Query and transform documents, images, and audio using SQL-powered pipelines
  • Schema-aware extraction: Automatically extract structured tables from unstructured documents like invoices and reports

Snowpark for custom processing

With Snowpark, you can:

  • Extract text from PDFs using Python
  • Perform image classification with embedded ML models
  • Parse JSON or log files into VARIANT columns
  • Run OCR, NLP, and generate embeddings via external functions
  • Build semantic search capabilities over document collections

VARIANT data type for semi-structured data

The VARIANT data type handles semi-structured data formats like JSON, XML, Parquet, and Avro:

  • Store complex, nested data structures
  • Query JSON fields directly using SQL
  • Maintain schema flexibility while preserving query performance

Why unified data architecture matters?

In most companies, data still lives in many places and tools. Dashboards sit on a legacy SQL warehouse, logs go to a separate observability stack, and documents and images disappear into unmanaged cloud buckets or shared drives.

Instead of stitching together a dozen point solutions, you can use Snowflake as the backbone of your data architecture and keep external systems only where they add unique value. The table below shows how data stack functions shift when you standardise on Snowflake in 2025:

Function Old architecture Snowflake in 2025
Analytics Separate SQL data warehouse Snowflake core engine
File storage S3, Google Cloud Storage, Azure Blob Internal storage plus external tables and integrations
Processing Spark clusters or ad hoc Python scripts Snowpark running in the same Snowflake account
Semi-structured & unstructured NoSQL database or object storage Native support in Snowflake tables and stages
Search & retrieval Elasticsearch or a separate search service Cortex search and vector search
ML & AI Separate ML platform and custom pipelines Snowflake AI Studio and Snowpark ML

Real-world use cases of handling unstructured data in Snowflake

Here is how this looks in practice. Below is our recent project, plus common patterns we see when teams bring documents, images, logs, and app data into Snowflake and put them to work.

Global finance, AI-ready in 90 days

A multinational finance firm spending more than 800K per month on cloud was battling rising costs and fragmented data. They needed a governed place for documents, logs, and tables. We used OpenFlow to ingest both structured and unstructured data into Snowflake, tracked lineage and policies in Horizon Catalog, set consistent business logic with semantic views, and enabled natural language querying through Cortex AI SQL. The result was about an 80% reduction in ingestion latency, real-time cost visibility with FinOps, and a platform ready for analytics, ML, and AI at scale.

Read how a global finance managed unstructured data in Snowflake →

Limitations and considerations of Snowflake

Snowflake’s unstructured data capabilities are strong, but it won’t fully replace your data lake or media platform. For B2B teams planning at scale, keep these practical constraints in mind:

  • Not a pure object storage replacement: Snowflake complements rather than replaces S3/GCS for massive-scale raw object storage
  • File retrieval performance: Binary object retrieval speed varies by file size and stage type
  • Compute costs: AI and ML workloads require careful resource management
  • Specialised use cases: For intensive video/audio editing, use specialised systems.

Best practices for managing unstructured data in Snowflake in 2025

1. Keep big binaries in external object storage, keep brains in Snowflake

Register S3, Blob, or GCS as external stages and reference files via the FILE type; keep only hot assets in internal stages for speed.

2. Standardize file layout and formats from day one

Use predictable paths (org/source/system/YYYY/MM/DD/id) and checksums; prefer compressed columnar formats like Parquet, with extracted text or page JSON beside PDFs and images.

3. Store metadata and embeddings in Snowflake, not in files

Put raw files in stages, but keep metadata, chunks, and embeddings in Snowflake tables linked by stable URIs for fast search and governance. Use directory tables to catalog staged files.

4. Orchestrate ingest → extract → enrich → index → serve with Snowpark

Run OCR, NLP, and parsers as Snowpark tasks and UDFs; batch, log runs, and make jobs idempotent so reruns are safe. Implementation flow in processing files with Snowpark.

5. Treat AI as a costed product

Separate warehouses for ELT and AI, strict auto-suspend, resource monitors, caching, and reuse of embeddings and summaries. Get a baseline with the FinOps savings calculator.

6. Govern at the row, column, and file edge

Classify on arrival, enforce row and column policies with masking, and keep least-privilege stage access and full lineage. For role design patterns, see Snowflake role hierarchy best practices.

Need a hand?

Our snowflake experts at Snowstack can audit your current setup, design a lean reference architecture, and prove value with a focused pilot. Read how we deliver in How we work or talk to a Snowflake expert.

Final thoughts

Snowflake doesn’t just store unstructured data; it makes it usable for search, analytics, and AI. With stages, the FILE data type, VARIANT, Snowpark, and Cortex, you can land documents, images, and logs alongside your tables, extract text and entities, generate embeddings, and govern everything under a single security and policy model. The winning pattern is simple: keep raw binaries in low-cost object storage, centralise metadata and embeddings in Snowflake, and start with one focused, high-value use case you can scale.

Ready to try this in your stack?

FAQs

Yes. Snowflake stores and processes unstructured files via stages (internal or external) and a FILE column type. You can access them with SQL and AI features. For setup help, see Snowflake implementation and AI and data governance.

Snowstack builds end-to-end pipelines for documents, images, logs, and app data. Start with Snowflake implementation or Contact.

A focused 4 to 6 week build: audit, reference architecture, secure stages and directory tables, ingest and extract jobs, embeddings and search, cost guards, and a demo with success metrics. See How we work.

FILE is a column type that holds a reference to a staged file (plus metadata like MIME type, size, etag, last modified, and URLs). It doesn't store the binary itself, just a pointer with metadata and helper functions (e.g., FL_GET_SIZE). We design schemas that use FILE in Advisory and architecture.

Create a stage, enable a directory table, then map staged files into a FILE column. We set this up during Migrations and integrations and Snowflake implementation.

Use internal stages for simplicity and hot paths. Use external stages when files live in S3, Azure Blob, or GCS. We help you choose in Advisory and architecture.

Use PUT to upload to internal stages, LIST to enumerate, and GET to download from internal stages. For external stages, upload with your cloud provider tools. At Snowstack, we standardise this in Migrations and integrations.

A directory table catalogs files on a stage so you can query, join to metadata, and build pipelines that react to file changes (with refresh/auto-refresh).

Yes. Use built-in services for document extraction, image understanding, and natural language queries. We enable safe usage through AI and data governance.

Yes. Snowflake provides a VECTOR data type, vector similarity functions, and embedding utilities for RAG/search over your files' text.

Aim for mid-sized files to balance parallelism and overhead; split very large files and compact many tiny ones. Get a sizing plan via Advisory and architecture.

Use scoped URLs (time-limited ~24h) or file URLs (require stage privileges). You can also generate scoped URLs with BUILD_SCOPED_FILE_URL.

Internal stage storage is billed by Snowflake; external stage storage is billed by your cloud provider; compute and any egress are separate. Start with the FinOps Savings Calculator and FinOps services.

Yes. Use a directory table (file catalog) and join it to tables holding metadata (e.g., owners, tags, PII flags) to power governance and pipelines.

Explore our latest blog posts for valuable insights.
View more insights
Stay up to date

Top data insights, delivered to your inbox

 Thanks for joining us!

We’ll keep you posted with fresh updates and resources.

Oops! Something went wrong while submitting the form.

Transform your data with Snowflake

You don't need to hire a data army or wait months to see results. Our Snowflake specialists will get you up and running fast, so you can make better decisions, cut costs, and beat competitors who are still stuck with spreadsheets and legacy systems

Learn more