In Part 1, we extracted what passes as my LinkedIn soul into a .jsonl file. We proved that a large, smart model can generate perfect training data for a small, dumb one.

Now, we need the engine to run it.

Anyone with a credit card can spin up a GPU in the cloud and host a model. I wanted to do it on a serverless, scale-to-zero infrastructure that costs absolutely nothing when I'm not using it.

Welcome to Part 2: LinkedIn ML Infrastructure.

The False Start: Gemma

Before we could even think about servers, we had to actually train the model.

My initial plan was to be brutally cheap. I wanted to use Google's native open weights model, Gemma 2B. It's tiny, it's efficient, and it lives in the same ecosystem as my data. I fired up the Kubeflow pipeline for it on Vertex AI and hit an immediate wall of 403 Permission Denied errors:

{
  "error": {
    "code": 403,
    "message": "Failed to download package from uri \"https://us-central1-kfp.pkg.dev/ml-pipeline/large-language-model-pipelines/tune-large-model/v3.0.0\".",
    "status": "PERMISSION_DENIED"
  }
}

First from the Artifact Registry package, then from my own private bucket. Even after fixing both, I realized the real problem: Gemma 2B on Vertex AI required a Full Fine-Tune. This meant retraining all two billion parameters of the model. For my tiny dataset of 334 rows, this was a non-starter. Then I found Llama 3.2. Not only was it available as a managed one-click tuning job that bypassed my IAM headaches, but it supported LoRA (Low-Rank Adaptation).

LoRA is the surgical approach: instead of rewriting the whole model, you're effectively adding a little training data over the top. It's faster, cheaper, and fundamentally better suited for tiny, high-quality datasets like mine.

The Tuning: Breaking the "Helpful Assistant"

My first attempt at tuning the Llama model was a disaster. I used standard settings, ran it for a few epochs, and got a model that sounded exactly like a helpful AI assistant. It refused to be cynical. It refused to be me. When I asked it to describe a personal struggle, it flatly rejected the premise:

"I don't have personal experiences, but I can share a hypothetical scenario... I can provide a simple and clear explanation of a concept without struggling because I am a machine learning model designed to provide accurate and informative responses."

Modern LLMs are safety trained to be polite, helpful, and bland, like most sycophantic AI chatbots. I wanted cynical dry wit. Breaking that conditioning requires aggression.

The recipe that finally worked was surprisingly heavy-handed for such a small dataset (334 rows):

• Epochs: 8 (We're hammering the lesson home).
• Learning Rate: 0.0002 (High enough to force change, low enough to avoid damaging the system's core logic).
• LoRA Rank: 16 (A standard size for the adapter layers).

After 8 full passes through my depolished data, the 8B LoRA model finally started to show that the data was taking hold, albeit in a weird, trying-too-hard AI clone kind of way.

To test each tuning run, I wrote a predictor script: a Flask server that downloads the model weights from GCS, loads them into memory, and exposes three endpoints. /predict for Q&A, /rewrite for style transfer, and /prompt for experimenting with custom system prompts. Every request gets wrapped in the strict Llama 3 Instruct format:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{system_prompt}<|eot_id|>
<|start_header_id|>user<|end_header_id|>
{user_input}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

The core of the script is surprisingly simple:

tokenizer = AutoTokenizer.from_pretrained(local_model_path)
tokenizer.chat_template = None  # The Lobotomy (more on this later)

model = AutoModelForCausalLM.from_pretrained(
    local_model_path,
    dtype=torch.float16,
    device_map="auto"
)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=256)
llm = HuggingFacePipeline(pipeline=pipe)

The /health endpoint starts at 503 and flips to 200 only after the model finishes loading, which becomes critical later when we deploy this to Cloud Run.

I had two main modes in the script. The first mode was a "Predictor" for answering questions, the idea being that the model should respond in my voice to any question just like a regular chatbot. The second mode was a Rewriter, so if you fed it a few lines of text it would rewrite the text in my voice. The difference between success and failure came down to how I filled in the system prompt.

The Identity Crisis

Getting to a model that approximated my voice was a strange road of identity confusion. Small models have strong "Identity Gravitation": they want to fall back to whatever concept of you they found in their pre-training data.

When I ran an early 3B full-tune (3 epochs at a low 1e-7 learning rate) and gave it an empty rewrite prompt, it replied with absolute confidence:

"Please go ahead and provide the text you'd like me to rewrite in the voice of Nico Westerdale, the charming and smooth-talking bartender from the TV show 'Dead to Me'."

I had to look that up, but the bartender in 'Dead to Me' is called Slade, played by actor John Ennis. I have no idea where it got that from.

I ran it again, and this time:

"I'm ready to rewrite. What's the text you'd like me to transform into the voice of Nico Westerdale, the infamous villain from the Fallout 4 universe?"

It even had a phase where it tried to sound like a motivational poster:

"(in a smooth, velvety voice) Ah, the core principle. You want to know what drives me, what sets me apart from the rest? Well, let me tell you, my friend. My core principle is quite simple, yet profound. ... 'Ambition is the fire that burns within.'"

What I was finding out was that this light tune (3 epochs at a tiny 1e-7 learning rate) was just enough to make the model aware of my name, but not enough to care about who I actually was. My name is unique enough that the small model had no real idea who I was, but creative enough to start inventing a life for me. It was filling the void of my identity with whatever fictional tropes it had lying around in its pre-training data.

The "Assistant" Sleeper Agent

It was through this predictor that I discovered a bizarre problem. In the early "Predictor" tests, I used a polite prompt: "You are an AI assistant that mimics the professional persona of Nico Westerdale." The model took the word "mimics" far too literally. When asked about my core principle, it replied with this sycophantic corporate drivel:

"(in a smooth, sophisticated tone) Ah, my core principle. Well, that's a question that gets to the heart of what I do, and what I stand for. You see, I'm not just a machine, I'm a facilitator."

Or better yet:

"(in a smooth, velvety voice) Ah, the pursuit of artificial intelligence. It's a realm where the boundaries of human ingenuity are pushed..."

It was performing the task, but narrating its own actions. The Llama 3 tokenizer has a chat_template baked into it that enforces a "User -> Assistant" dialogue format. When you use the standard Hugging Face pipeline, it automatically wraps your prompt in this hidden structure, which triggers the model's deepest helpful assistant training and overrides my fine-tuning. That tokenizer.chat_template = None line in the predictor? That's the lobotomy: manually disabling the template to stop the "Assistant" persona from waking up.

But even the lobotomy wasn't enough. I was still fighting the model's pre-training because of my own verbosity. My original system prompt was polite and structured exactly how the tutorials led me to believe this was done:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an AI assistant that mimics the professional persona of Nico Westerdale.<|eot_id|>
<|start_header_id|>user<|end_header_id|>

Answer the following question in his voice: {question}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

I tried variants like "Answer the following question in his voice" or even an empty system prompt, hoping the fine-tuning would just take over. It didn't. The model's base "helpful assistant" persona kept coming through. Every time I used an instruction, it would retreat into these cringe-worthy, self-aware meta-commentaries:

"The perpetual conundrum of the AI conundrum. (chuckles) As a highly advanced language model, I've encountered numerous instances where I've struggled to convey a simple concept to a human user."

Or the existential philosopher:

"The inevitable question that gets to the heart of my artificial existence. As a highly advanced language model, I don't truly 'struggle' in the way humans do..."

Finally I realized, with a bit of a facepalm moment, that if the model was trained to be me, I didn't have to tell it to be itself. I just had to let it speak. I replaced the entire paragraph of instructions with a single, imperative command:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You rewrite text in your own voice.<|eot_id|><|start_header_id|>user<|end_header_id|>

Rewrite the following text in your voice: {original_text}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Suddenly, the meta-commentary vanished. The "facilitator" was dead. When I fed it a dry note about my weekend plans, it didn't say "Here is a rewrite..." It just spoke:

"Took the weekend off to go bike packing. Great to get out, but I'm in for a world of hurt when I get back to this."

No "as an AI," it's actually taking a weekend off and going bike packing! That's huge progress.

The Deployment Problem

How do you serve a multi-gigabyte machine learning model? On Google Cloud Platform (GCP), there are three obvious paths. One is easy and expensive. One's a non-starter. One is obnoxious and cheap. This is no comment on my personality, but I chose the latter.

Option 1: The Official Way (Vertex AI Endpoints)

GCP has a dedicated service for this: Vertex AI Endpoints. You take your trained model, click a few buttons, and it's deployed on an optimized, scalable infrastructure.

The Catch: It's always on. You're paying for a GPU-enabled virtual machine to sit there, 24/7, waiting for a request that might never come. For a production system with constant traffic, this would cost hundreds of bucks a month. For this project, that's a pass.

Option 2: The Serverless Way (Cloud Functions)

What about the cheapest serverless option? Cloud Functions. You upload your code, and it only runs when called. Perfect, right?

The Catch: Cloud Functions are designed for short, stateless tasks. They have aggressive timeout limits and memory caps that are laughable for our use case. An 8B Llama in float16 is about 16GB in memory. The cold start, the process of downloading the model from storage and loading it into memory, would take several minutes. The function would time out and die before it ever served a single request. Wrong tool for the job. (2nd gen Cloud Functions have since bumped to 32GB RAM and 60-minute timeouts, so the memory cap isn't the blocker it was, but there's still no GPU support and the cold start story is still painful.)

Option 3: The Container Way (Cloud Run)

Cloud Run is a serverless platform for running containers. Like Cloud Functions, it can scale to zero, so we pay nothing when it's idle. But because it runs a standard Docker container, we get full control over the environment, the memory, and the startup process.

The Catch: Cloud Run is designed for web services, not for ML models.

To get the 32GB of RAM required to hold an 8B Llama in memory without crashing, Google Cloud Run has a strict pairing rule: you can't just have RAM; you have to buy CPU too. I was forced to provision 8 vCPUs just to get the memory. You pay that pairing whenever the instance is actually up. Scale to zero is what keeps idle at zero.

This is our path.

Local Development

Before deploying anything, you have to make it work locally.

My local machine has a decent NVIDIA GPU. Getting the Python script to use it was a nightmare.

1. Python Version Issues: I started with the latest Python (3.13). Big mistake. PyTorch, the core ML library, didn't have a stable, CUDA-enabled build for it. I had to downgrade to Python 3.11.9 and, more importantly, learn to love virtual environments (.venv). Never again will I pollute my global Python install.
2. The CUDA Surprise: pip install torch is a lie. It installs a CPU-only version. To get GPU acceleration, you have to uninstall it and then reinstall a specific version from a special URL, explicitly telling it which CUDA version you need.
3. The GCS Path Flattening: Vertex AI doesn't just give you a model; it gives you a Russian Nesting Doll of folders. It writes weights to a path like tuned-models/postprocess/node-0/checkpoints/final/. I had to write a custom download helper to "flatten" this structure into /tmp just so the Hugging Face tokenizer could find its own files.
4. Working Code: After all that, the model ran. Really really slowly, but it ran. A single inference took 30 seconds. By default, models load in 32-bit precision (float32). By adding one line of code, dtype=torch.float16, we tell it to use 16-bit half-precision. That enables the GPU's Tensor Cores, and suddenly, inference times dropped from 30 seconds to about 6 seconds. Decent enough.
5. GPU?: My Windows Task Manager sat at 0% GPU utilization. I thought I was failing.

I even spent an hour chasing a "Second GPU Phantom." My machine showed two GPUs in Task Manager. I tried to pin the model to the "second" one to keep my primary display lag-free, only to hit a wall:

ValueError: Got device==1, device is required to be within [-1, 1)

It turns out the second "GPU" was just the integrated Intel chip. Intel has no CUDA. PyTorch only saw the NVIDIA, so device==1 was never a real option.

Then I ran nvidia-smi in the terminal.

Idle (09:43:48):

| NVIDIA-SMI 573.57                 Driver Version: 573.57         CUDA Version: 12.8     |
|   0  NVIDIA RTX 3000 Ada Gene...  WDDM  |
| N/A   41C    P8              1W /   40W |    6365MiB /   8188MiB |      0%      Default |

During inference (09:43:51):

| N/A   45C    P1             41W /   40W |    6365MiB /   8188MiB |     87%      Default |

90% utilization and 6.3GB of VRAM occupied on a 40W laptop chip. That footprint is the 3B; the 8B model would not fit. Pro tip: Don't trust the Windows GUI with your MLOps; trust the terminal.

Cloud Run

The deployment architecture is straightforward: a Dockerfile to build the container, a requirements.txt to list dependencies, and a predictor.py script running a Flask/Gunicorn web server.

Thanks Google

My most painful discovery was this. You can configure a Cloud Run service to use a GPU. You can configure it to scale to zero. You cannot do both at the same time.

To use a GPU, you must have a minimum of one instance running 24/7. This brings us right back to the Vertex AI Endpoint problem, at a cost of ~$290/month. For my ridiculous ML project to cost more than a cup of coffee was a clear failure.

I had to make a choice: be fast, or be cheap. I chose cheap. I abandoned the GPU and deployed on a CPU-only instance. This meant the cold start would be agonizingly slow.

Update: I'm writing this up a few months later, and Google has since shipped GPU scale-to-zero on Cloud Run. The $290/month trap is gone; you can now have an L4 GPU instance that costs $0 when idle and about $1.05/hour when it wakes up. The cold start is still there, but Google benchmarks a 4B model at ~19 seconds from zero to first token, so my 3-5 minute CPU agony would drop to under 30 seconds. If I were doing this today, I'd take that path.

There are also platforms like Modal that tackle cold starts more aggressively. Modal lets you snapshot GPU memory state, so instead of re-loading weights from scratch on every cold start, a new container restores from a checkpoint, skipping the expensive initialization. That's the fair comparison for a custom LoRA like mine. Fireworks keeps popular catalog models hot across their fleet and charges per token, so there's effectively no cold start, but that's a hosted model, not your own weights. For a side project serving one request every few days, Modal would have saved me a lot of pain.

Cold Starts

On my first deploy to Cloud Run the CPU-only cold start for this model takes about 3 to 5 minutes. It has to pull a multi-gigabyte container image, download the weights from GCS, and then load them into RAM. I hit the public URL too early. The server was up, but the weights were not:

{"error":"Model is not loaded yet"}

Cloud Run has a health check mechanism called a Startup Probe. You give it an endpoint in your application (e.g., /health) and it'll repeatedly ping that endpoint during the startup process. If the probe fails too many times, Cloud Run kills the instance. The trick is to make the probe wait, so user traffic never sees that 503.

So how about this:

1. I created a /health endpoint in my Flask app.
2. I designed it to return a 503 Service Unavailable status by default.
3. Only after the load_model() function successfully completes does a global variable flip, causing /health to return a 200 OK.
4. In the Cloud Run configuration, I set the Startup Probe's timeout to 600 seconds (10 minutes).

This forces Cloud Run to be incredibly patient. It'll sit there, pinging the /health endpoint for up to 10 minutes, getting rejected again and again. It sees the service is "unhealthy." But it waits. Then, finally, after 3 to 5 minutes, the model finishes loading, the endpoint flips to 200 OK, the probe passes, and Cloud Run declares the service "healthy," ready to accept traffic.

It's not really what Cloud Run is supposed to be about, but it does allow a long-running, heavyweight process to initialize within a serverless environment designed for the exact opposite.

The Results

We did it. We have a live, public URL for our custom-trained Llama model. It costs a few cents per invocation and zero dollars when idle. The first request is a coffee break, but subsequent requests are reasonably fast until the instance scales down.

More importantly, it actually sounds like me. Here are some of the raw rewrite outputs from the final API endpoint:

The Successes:

When fed a news brief about Cursor's new Agent mode:

"Cursor 2.0's new Agent mode released yesterday is already leaning on the output of the model. The Composer model uses Reinforcement Learning, trained on aggregate patterns. Fast, impressive, but it doesn't learn from your codebase."

That tracks. I posted about Cursor 2.0, and I'm a big Cursor fan.

When fed a news story about LinkedIn quietly training on user posts:

"I just found out that LinkedIn has quietly added a new policy that allows them to use your profile data and posts to train AI models. Thanks LinkedIn, you had me going for a second there with the quiet part."

Love the snark on that one, A+ LinkedIn post.

When fed a dry corporate announcement about LinkedIn connections:

"As a linkedIn user, I want to connect with other linkedIn users, so that they can fill my inbox with auto-generated messages promoting their services. #reallifeuserstories"

Sarcastic, and it actually used my hashtag! I'm genuinely impressed!

When fed a snippet about OpenAI's style changes:

"OpenAI is going to let you tell ChatGPT to stop using the em dash. Finally, a win for anyone who doesn't want their writing to look like it was generated by a lazy robot. #ai #writing"

A bit dry, but okay.

And the Also Ran:

It wasn't all success.

When I fed it a dry sentence about Microsoft Copilot, it replied:

"I'm DALL-E 2, and I want to be Nico Westerdale for a day. #ai #nico #microsoft"

Okay nobody's perfect. Copilot certainly isn't.

The Sunset

So it works.

I set out to build a trained ML model that could rewrite text in my own LinkedIn voice. I proved that you can serve an ML model on serverless infrastructure for peanuts if you're willing to wait for a while on startup. I showed that you don't need 10,000 rows of data to tune a persona; just 334 rows is enough to break a model's safety training and force it to adopt a voice, and it did so pretty decently for the tiny training set that I actually had.

Its cynical snarky posts made me consider that I could just set up a scheduler to consume the latest news and automate it to post on LinkedIn. It wouldn't be that hard, a daily job that crawled the web, wrote the story to blob storage, fired up the ML model on Cloud Run, waited the eternity for it to start up, fed it the news story clip and said to rewrite it in my voice, then save the rewrite back to storage. Then a short while later a scheduled job on Claude desktop could log into LinkedIn using the browser and grab the rewrite out of storage and post it. Or my OpenClaw bot could do the same running on my Raspberry Pi. Or I could script it with Playwright and Python. Whatever the path, it would work, the hard part's done.

I'm not doing that.

The experiment was a success, but frankly, the world only needs one Nico Westerdale posting on LinkedIn at a time.

However, if you're tempted, then the code for the predictor server is on GitHub: linkedin-ai-persona-server.