Pinecone: Cloud Serverless Vector Search

1. Pinecone Serverless vs. Pod-Based Architecture

Historically, vector databases ran on dedicated virtual machine "pods" provisioned with fixed CPU and RAM.Pinecone Serverless revolutionized cloud vector storage by cleanly separating storage from compute:

2. Installation & Python SDK v5+ Setup

Install the official Pinecone client:

pip install pinecone-client

Initialize the client using your API key from the Pinecone Cloud Console:

import os
from pinecone import Pinecone, ServerlessSpec

# Initialize Pinecone Client
pc = Pinecone(api_key=os.environ.get("PINECONE_API_KEY", "your-api-key"))

# List existing cloud indexes
print("Existing Indexes:", [idx.name for idx in pc.list_indexes()])

3. Creating a Serverless Index

Define the index dimensionality (e.g., 1,536 for OpenAI embeddings), distance metric (cosine, euclidean, or dotproduct), and the cloud region using ServerlessSpec:

INDEX_NAME = "enterprise-rag-serverless"
DIMENSION = 1536

# Check if index already exists
if INDEX_NAME not in [idx.name for idx in pc.list_indexes()]:
    print(f"Creating serverless index '{INDEX_NAME}'...")
    pc.create_index(
        name=INDEX_NAME,
        dimension=DIMENSION,
        metric="cosine",
        spec=ServerlessSpec(
            cloud="aws",
            region="us-east-1"
        )
    )
    print("Index provisioned successfully!")

# Connect to the target index
index = pc.Index(INDEX_NAME)
print("Index Host URL:", index.describe_index_stats())

4. Enterprise Multi-Tenancy with Namespaces

In SaaS applications, you must strictly prevent Tenant A from accessing Tenant B's data. Creating separate vector indices for every user is prohibitively expensive. Pinecone solves this via Namespaces:

# Upserting batched vectors into isolated tenant namespaces
sample_vectors = [
    {
        "id": "kb_doc_101",
        "values": [0.012] * 1536,  # 1536-dimensional float vector
        "metadata": {
            "title": "Refund Policy 2024",
            "department": "billing",
            "access_level": "public",
            "chunk_text": "Customers can request a full refund within 30 days of purchase."
        }
    },
    {
        "id": "kb_doc_102",
        "values": [0.045] * 1536,
        "metadata": {
            "title": "SOC2 Compliance Guidelines",
            "department": "security",
            "access_level": "confidential",
            "chunk_text": "All employee laptop drives must have FileVault encryption enabled."
        }
    }
]

# Upsert into specific tenant namespace
index.upsert(
    vectors=sample_vectors,
    namespace="tenant_acme_corp"
)

print("Upsert completed for tenant_acme_corp!")

5. Low-Latency Querying with Metadata Filters

Query the vector index by supplying the embedded query vector, setting include_metadata=Trueto retrieve document text directly, and passing a JSON metadata filter:

# Simulated query vector (e.g. encoded from "How do I get my money back?")
query_vector = [0.015] * 1536

# Execute top-K vector search with strict metadata filtering
query_response = index.query(
    vector=query_vector,
    top_k=3,
    namespace="tenant_acme_corp",
    include_metadata=True,
    filter={
        "department": {"$eq": "billing"},
        "access_level": {"$in": ["public", "internal"]}
    }
)

print("=== Pinecone Query Results ===")
for match in query_response["matches"]:
    print(f"ID: {match['id']} | Score: {match['score']:.4f}")
    print(f"Title: {match['metadata']['title']}")
    print(f"Text Snippet: {match['metadata']['chunk_text']}\n")