Saturday, June 21, 2025

Table types exercise

1.    Permanent Table:

Use Case: Storing customer data for long-term use.

 

-- Create table

CREATE OR REPLACE TABLE customers (

    customer_id INT,

    name STRING,

    email STRING,

    signup_date DATE

);

 

-- Insert data

INSERT INTO customers VALUES

    (1, 'Alice', 'alice@example.com', '2024-01-15'),

    (2, 'Bob', 'bob@example.com', '2024-02-20');

 

-- Query with Time Travel (retrieve data as of 1 hour ago)

SELECT * FROM customers AT(OFFSET => -3600);

 

2.    Temporary Table

Use Case: Storing intermediate results during a session.

-- Create temp table for active users
CREATE TEMP TABLE active_users AS
SELECT * FROM customers 
WHERE signup_date >= '2024-01-01';
 
-- Query temp table (only visible in this session)
SELECT * FROM active_users;
-- No need to drop; vanishes when session ends.

3. Transient Table

Use Case: Staging data for ETL (no Fail-safe needed).

-- Create transient table
CREATE TRANSIENT TABLE raw_logs (
    log_id INT,
    event_time TIMESTAMP,
    event_data VARIANT
);
 
-- Load data (e.g., from a stage)
COPY INTO raw_logs FROM @my_stage/logs/;
 

4. External Table

Use Case: Querying Parquet files in S3 without loading.

-- Create external stage (if not exists)
CREATE STAGE my_s3_stage
  URL = 's3://my-bucket/path/'
  CREDENTIALS = (AWS_KEY_ID = '...' AWS_SECRET_KEY = '...');
 
-- Create external table for Parquet files
CREATE EXTERNAL TABLE sales_external
  LOCATION = @my_s3_stage/sales/
  FILE_FORMAT = (TYPE = PARQUET);
 
-- Query directly
SELECT * FROM sales_external WHERE region = 'EMEA';

 

5. Dynamic Table

Use Case: Auto-refreshing aggregated sales data.

 

-- Source table (orders)
CREATE TABLE orders (
    order_id INT,
    customer_id INT,
    amount NUMBER(10,2),
    order_date DATE
);
 
-- Dynamic table (materialized view)
CREATE DYNAMIC TABLE daily_sales
  TARGET_LAG = '5 MINUTES'  -- Refresh every 5 mins
  AS
  SELECT 
    order_date,
    SUM(amount) AS total_sales
  FROM orders
  GROUP BY order_date;
 
-- Query (always up-to-date)
SELECT * FROM daily_sales;
 

6. Iceberg Table

Use Case: Interoperability with data lakes.

-- Create external volume (if not exists)
CREATE EXTERNAL VOLUME iceberg_vol
  STORAGE_LOCATIONS = (
    (
      NAME = 'my-iceberg-bucket'
      STORAGE_PROVIDER = 'S3'
      STORAGE_BASE_URL = 's3://my-iceberg-bucket/'
    )
  );
 
-- Create Iceberg table
CREATE ICEBERG TABLE customer_iceberg
  CATALOG = 'my_catalog'
  EXTERNAL_VOLUME = 'iceberg_vol'
  AS SELECT * FROM customers;
 
-- Query (supports Iceberg features like time travel)
SELECT * FROM customer_iceberg VERSION AS OF '2024-05-01';

 

Key Operations Across All Tables

-- View table metadata
SHOW TABLES LIKE '%customers%';
 
-- Clone a table (zero-copy)
CREATE TABLE customers_dev CLONE customers;
 
-- Drop a table
DROP TABLE customers_dev;

 

Advanced Snowflake Table Examples with Real-World Scenarios

1.   Permanent Table with Advanced Features

Use Case: Enterprise customer data warehouse with change tracking

 

-- Create a permanent table with clustering and comment
 
CREATE TABLE customers (
    customer_id INT AUTOINCREMENT START 1 INCREMENT 1,
    name STRING NOT NULL,
    email STRING UNIQUE,
    signup_date DATE DEFAULT CURRENT_DATE(),
    lifetime_value NUMBER(38,2),
    preferences VARIANT,
    created_at TIMESTAMP_LTZ DEFAULT CURRENT_TIMESTAMP(),
    updated_at TIMESTAMP_LTZ DEFAULT CURRENT_TIMESTAMP(),
    COMMENT = 'Master customer dimension table'
) CLUSTER BY (date_trunc('MONTH', signup_date));
 
-- Add a tag for PII classification
ALTER TABLE customers SET TAG security.pii_level = 'high';
 
-- Create a stream for change tracking
CREATE STREAM customer_changes ON TABLE customers;
 
-- Create a task to process changes
CREATE TASK process_customer_updates
  WAREHOUSE = compute_wh
  SCHEDULE = '5 MINUTE'
AS
  MERGE INTO customer_archive t
  USING (SELECT * FROM customer_changes) s
  ON t.customer_id = s.customer_id
  WHEN MATCHED THEN UPDATE SET t.updated_at = CURRENT_TIMESTAMP()
  WHEN NOT MATCHED THEN INSERT (customer_id, name, email) 
  VALUES (s.customer_id, s.name, s.email);

 

 

 

 

 

Snowflake Table Types

 "In my recent banking project on Snowflake, I used different table types for optimized performance and cost.

I used External Tables to read raw customer files directly from S3.
Then I staged them in Transient Tables for cleansing.
Cleaned data was loaded into Permanent Tables like customer_master.
For analytics, I followed a star schema using fact and dimension tables.
To speed up dashboards, I created Materialized Views for monthly summaries and also used Clone Tables during testing and validation.
This structure allowed me to build a robust, cost-efficient, and scalable banking analytics pipeline."

Snowflake Table Types - Interview Summary

1. Table Types & Use Cases

  • Permanent Table – For storing validated business data (e.g., customer_master)

  • Transient Table – For intermediate data (e.g., raw_txn_data_trans); no Fail-safe

  • Temporary Table – Used within session only

  • External Table – Reads raw files from cloud (e.g., S3)

  • Materialized View – Stores pre-aggregated data for performance

  • Clone Table – Fast, zero-copy clone for testing/validation


2. Layered Table Usage in Retail Banking

  • 🔹 Raw Zone – raw_customer_data_ext, raw_txn_data_trans

  • 🔹 Clean Zone – customer_master, account_master

  • 🔹 Analytics Layer – txn_fact, product_dim, customer_dim

  • 🔹 Reporting Layer – monthly_txn_summary_mv, top_branches_mv

Friday, June 20, 2025

Data Engineering rounds 123

 

Round 1: SQL + Problem Solving

  1. Find 2nd latest transaction per user


SELECT user_id, transaction_date FROM ( SELECT user_id, transaction_date, ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY transaction_date DESC) AS rn FROM transactions ) t WHERE rn = 2;
  1. Rank records inside partitions

Use RANK() or ROW_NUMBER():

SELECT user_id, transaction_amount,
RANK() OVER (PARTITION BY user_id ORDER BY transaction_amount DESC) AS rnk FROM transactions;
  1. Explain why NULLs break NOT IN

If even one NULL is in the list, NOT IN returns no results:

SELECT 1 WHERE 10 NOT IN (1, 2, NULL); -- returns nothing

Reason: NULL means unknown, so 10 != NULL cannot be evaluated, making the result unknown, not true.

Fix: Use NOT EXISTS or filter NULLs first.


Round 2: Data Pipeline Design

  1. Build a pipeline for 100M+ daily events

Key Points:

  • Use a distributed system: Kafka (ingestion), Spark/Beam (processing), S3/GCS or HDFS (storage), and Snowflake/BigQuery (warehouse).

  • Partition data by date/hour, compress with Parquet/ORC.

  • Optimize for scalability and retries.

  • Schedule with Airflow or Dagster.

  1. What if one batch fails mid-way?

  • Implement checkpointing.

  • Use retry logic in orchestrator (Airflow).

  • Store raw data and reprocess from failure point.

  • Use idempotent writes to avoid duplicates.

  1. Schema drift or late-arriving data

  • Use schema registry and schema evolution tools.

  • Store raw data separately for reprocessing.

  • Use late windowing in Spark or Dataflow.

  • Add versioning and data quality validation.


Round 3: System Design / Architecture

  1. Design a scalable data lake

  • Use cloud storage (S3/GCS).

  • Partition data (by date/event type).

  • Use Delta Lake or Apache Iceberg for versioning, ACID.

  • Add metadata catalog (Glue, Hive Metastore).

  • Use Spark or Presto for querying.

  • Include monitoring, retries, alerting.

  1. Track global ride data in real time

  • Use event-driven architecture:

    • Kafka → Stream processor (Flink/Spark Structured Streaming) → NoSQL DB (Cassandra/Redis) or warehouse.

  • Low-latency writes and geo-partitioned storage.

  1. Tradeoffs of batch vs stream
    | Criteria | Batch | Streaming |
    |--------------|------------------------------------|------------------------------------|
    | Latency | High (hours/minutes) | Low (seconds/milliseconds) |
    | Complexity | Easier | Harder (state mgmt, ordering) |
    | Cost | Lower | Higher (always-on infra) |
    | Use Cases | Reporting, historical analysis | Fraud detection, real-time alerts |


Round 4: Stakeholder & Behaviorals

  1. Explain a failed pipeline to a non-tech manager

“We had an issue where part of the data wasn’t loaded due to a missed file drop from the source system. We implemented automated checks to alert us earlier and ensure retry happens immediately, so downstream reports stay accurate.”

  1. Time your data was wrong

“Once we onboarded a new source system, we misaligned date formats, causing incorrect reporting. I ran data audits, corrected the logic, and set up validation checks to prevent this in future pipelines.”

  1. Can analysts trust what you build?

“Yes — I prioritize data quality, version control, logging, and transparency. I document data sources, add unit tests, and implement alerting and data freshness checks to ensure reliability.”


Thursday, June 19, 2025

Snowflake

 

1. Snowflake Basics

Q1: What is Snowflake, and how is it different from traditional databases?

Answer:

  • Snowflake is a cloud-based data warehouse built for scalability, performance, and ease of use.

  • Key differences:

    • Separates storage and compute (unlike traditional databases).

    • Auto-scaling (no manual sharding or partitioning).

    • Supports semi-structured data (JSON, Avro, Parquet) natively.

    • Zero-copy cloning (instantly duplicate databases/tables without extra storage).

    • Time Travel (recover deleted/modified data).


2. Setting Up Snowflake

Q2: How do you create a database, table, and warehouse in Snowflake?

Answer:

Create a Database

sql
Copy
Download
CREATE DATABASE sales_db;

Create a Table

sql
Copy
Download
CREATE TABLE customers (
    customer_id INT PRIMARY KEY,
    name VARCHAR(100),
    email VARCHAR(100),
    signup_date DATE
);

Create a Virtual Warehouse (Compute Cluster)

sql
Copy
Download
CREATE WAREHOUSE analytics_wh
WITH 
    WAREHOUSE_SIZE = 'X-SMALL'
    AUTO_SUSPEND = 300  -- Pauses after 5 mins of inactivity
    AUTO_RESUME = TRUE;

3. Data Loading & Querying

Q3: How do you load CSV and JSON data into Snowflake?

Answer:

Loading CSV Data

sql
Copy
Download
-- 1. Create a stage (cloud storage reference)
CREATE STAGE my_csv_stage
FILE_FORMAT = (TYPE = 'CSV' SKIP_HEADER = 1);

-- 2. Upload file to stage (via SnowSQL or UI)
-- 3. Load into table
COPY INTO customers
FROM @my_csv_stage/customers.csv;

Loading JSON Data

sql
Copy
Download
-- 1. Create a table with VARIANT column
CREATE TABLE json_data (
    raw_data VARIANT
);

-- 2. Load JSON file
COPY INTO json_data
FROM @my_stage/data.json
FILE_FORMAT = (TYPE = 'JSON');

Querying JSON Data

sql
Copy
Download
SELECT 
    raw_data:customer_id::INT AS customer_id,
    raw_data:address.city::STRING AS city
FROM json_data;

4. Advanced Querying & Optimization

Q4: How do you optimize query performance in Snowflake?

Answer:

Best Practices:

✅ Use clustering keys (for large tables):

sql
Copy
Download
ALTER TABLE large_table CLUSTER BY (date_column);

✅ Materialized views (for frequent queries):

sql
Copy
Download
CREATE MATERIALIZED VIEW mv_sales_summary AS
SELECT product_id, SUM(amount) 
FROM sales 
GROUP BY product_id;

✅ Warehouse scaling (for heavy workloads):

sql
Copy
Download
ALTER WAREHOUSE analytics_wh SET WAREHOUSE_SIZE = 'LARGE';

✅ Query caching (Snowflake caches results for 24 hours).


5. Data Sharing & Security

Q5: How do you share data securely in Snowflake?

Answer:

Secure Data Sharing (Provider-Consumer Model)

sql
Copy
Download
-- Provider creates a share
CREATE SHARE sales_data_share;

-- Grants access to objects
GRANT USAGE ON DATABASE sales_db TO SHARE sales_data_share;
GRANT SELECT ON TABLE sales_db.public.customers TO SHARE sales_data_share;

-- Adds consumer account
ALTER SHARE sales_data_share ADD ACCOUNTS = 'consumer_account';

Row-Level Security (RLS)

sql
Copy
Download
-- Create a policy to restrict data access
CREATE ROW ACCESS POLICY region_filter
AS (region VARCHAR) RETURNS BOOLEAN ->
    CURRENT_ROLE() = 'ADMIN' OR region = CURRENT_REGION();

6. Time Travel & Cloning

Q6: How do you recover deleted data using Time Travel?

Answer:

Restore a Dropped Table

sql
Copy
Download
-- Check dropped tables
SHOW TABLES HISTORY LIKE 'customers';

-- Restore the table
UNDROP TABLE customers;

Query Historical Data

sql
Copy
Download
-- See data from 2 hours ago
SELECT * FROM customers AT(OFFSET => -60*60*2);

-- Restore to a previous state
CREATE TABLE customers_restored AS
SELECT * FROM customers BEFORE(TIMESTAMP => '2024-01-01 12:00:00'::TIMESTAMP);

Zero-Copy Cloning

sql
Copy
Download
-- Clone a table (no storage cost)
CREATE TABLE customers_dev CLONE customers;

-- Clone an entire database
CREATE DATABASE sales_dev CLONE sales_db;

7. Snowflake Administration

Q7: How do you monitor costs and usage?

Answer:

Check Warehouse Usage

sql
Copy
Download
SELECT * 
FROM SNOWFLAKE.ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY
ORDER BY START_TIME DESC;

Monitor Storage Costs

sql
Copy
Download
SELECT * 
FROM SNOWFLAKE.ACCOUNT_USAGE.STORAGE_USAGE;

Set Budget Alerts

sql
Copy
Download
-- Use Snowflake's resource monitors
CREATE RESOURCE MONITOR budget_monitor
WITH CREDIT_QUOTA = 100
TRIGGERS 
    ON 80 PERCENT DO NOTIFY
    ON 100 PERCENT DO SUSPEND;

functions