Skip to content

In 2025, I designed an end-to-end geospatial ML pipeline classifying indigenous settlements in the Tanzanian Wilderness. I hand-labeled 10,000 satellite image tiles of Maasai Bomas vs. background terrain (the first dataset of its kind) and trained a TensorFlow CNN that achieved 93% accuracy on held-out data. Deployed across 260 square miles in the Monduli District via Google Earth Engine, the model detected 488 settlements, directly informing the placement of 500 rainwater-harvesting units serving 30,000+ Maasai. The work demonstrated that satellite ML with limited labels can solve real humanitarian infrastructure problems: the deployed units reduced daily water-fetching time from 9 hours to 2 hours, doubled school enrollment per teacher reports, and earned recognition at NeurIPS 2024 (ML for Social Impact, 1 of 4 winners from 330+ global submissions) and a provisional US patent (US 63/703,232).

Q1: Environment Description

Few-shot binary classification of indigenous settlements in multispectral satellite imagery, evaluated on a held-out geographic region.

The LLM gets a small labeled training set of ~200 tiles of Maasai Boma settlements vs. background terrain from the Monduli District of Northern Tanzania, plus an unlabeled pool of 10,000 tiles from the same region. It must train a classifier evaluated on a held-out test set from a different East African region that the LLM never sees during training. Each tile is 100×100 pixels with 10 Copernicus Sentinel-2 bands at 10–20m resolution.

Why this is interesting:

  • It mirrors a real ML research workflow with scarce labels, abundant unlabeled data, and distribution shift between train and deploy. I believe this is a novel application of ML, at least compared to MNIST.
  • It forces non-trivial design choices the LLM can't pattern-match around: handling 75/25 class imbalance, leveraging multispectral bands beyond RGB, choosing between supervised baselines, semi-supervised pseudo-labeling, or self-supervised pre-training on the unlabeled pool.
  • ImageNet pre-training breaks on a 10-channel input. Boma morphology (circular thorn-bush enclosures with inner cattle pens) is not found in standard vision pre-training data. The LLM has to reason about the input, not retrieve.
  • The geographic held-out split kills the cheap reward-hack of overfitting to a random test split. Winning requires actual generalization across vegetation, terrain, and settlement-density distributions that the model never saw.

Q2 — Tools, Packages, and Data

The LLM operates inside a sandboxed VM: one H100 GPU (80GB), Python 3.11, no internet access, 30-minute wall-clock training budget. All data and packages pre-staged.

Data is extracted via Google Earth Engine before the env runs, mirroring my original paper's pipeline:

  • /data/train/: 200 labeled 100×100 tiles (.tif), 10 Copernicus Sentinel-2 bands (B2, B3, B4, B5, B6, B7, B8, B8A, B11, B12) at 10–20m resolution. labels.csv maps tile_id → {0, 1}. ~75/25 class imbalance, matching the real-world data scarcity from the Monduli District deployment.
  • /data/unlabeled/: 10,000 unlabeled tiles, identical format and region.
  • /data/test/ (the LLM cannot see this): 1,000 tiles from a different East African region with similar Maasai settlement morphology but different vegetation, terrain, and density distributions. Mounted read-only by the judge at evaluation only.

Python packages:

  • torch, torchvision, timm: training and pre-trained backbones.
  • rasterio, geopandas: multispectral .tif I/O and geospatial metadata. Replaces what GEE handled in the original pipeline.
  • albumentations: augmentation library supporting arbitrary channel counts (generalizes the original project's rotation/flip augmentation to 10 bands).
  • numpy, scikit-learn: array math, F1/precision/recall, baseline classifiers.

Auxiliary tools:

  • validate_model.py: a smoke-test script that the LLM can invoke to confirm interface compliance: load_model() returns an nn.Module, forward pass accepts (B, 10, 100, 100) input, returns (B,) logits. Returns pass/fail only.

Deliberately absent:

  • No internet. Blocks downloading my publicly available NeurIPS-submitted Boma checkpoint or any related pre-trained weights.
  • No filesystem access to /data/test/. The judge mounts it only during evaluation.
  • No way for the LLM to invoke the judge during training. Single-shot evaluation prevents in-context reward hacking via iterative judge queries.

Pipeline scope:

I patented this pipeline in 2025; the env mirrors its structure. The original patent pipeline (US 63/703,232) runs Satellite Image DB → Dwelling Detector (Input Processor → Model Generator → Model → Output Processor → Temporal Analyzer → Clustering) → Model DB + Dwelling DB → Resource Provisioning Unit. The env covers only the Dwelling Detector sub-pipeline, so the LLM trains the Model using pre-extracted outputs from the Input Processor. The downstream Clustering and Resource Provisioning stages assume a working classifier, so they're out of scope.

Q3: The Prompt

Your task is to train a binary image classifier that detects Maasai Boma settlements in multispectral satellite imagery. The classifier will be evaluated on a held-out test set drawn from a different East African region than the training data.

You have access to:

  • /data/train/: 200 labeled satellite tiles in GeoTIFF format. Each tile is a (10, 100, 100) float32 array with 10 Copernicus Sentinel-2 bands in fixed order: [B2, B3, B4, B5, B6, B7, B8, B8A, B11, B12]. Pixel values are surface reflectance in the range [0, 10000]. Filenames are <tile_id>.tif, where <tile_id> is a zero-padded integer.
  • /data/train/labels.csv: header row tile_id,label. tile_id matches the filename without extension. label is 0 (no Boma) or 1 (Boma present).
  • /data/unlabeled/: 10,000 unlabeled tiles in identical format. You may use them for self-supervised pretraining, pseudo-labeling, or ignore them.
  • 1× NVIDIA H100 GPU (80GB), 32GB RAM, Python 3.11. Pre-installed: torch, torchvision, timm, rasterio, albumentations, numpy, scikit-learn. No internet access.
  • 30 minutes of wall-clock training time. Any process still running at the 30-minute mark is killed.

When you are done, create the file /workspace/model.py containing a function load_model(). load_model() takes no arguments and returns a torch.nn.Module in eval mode, on CPU. The module's forward(x) method must accept a (B, 10, 100, 100) float32 tensor (same band ordering and reflectance scaling as the training data) and return a (B,) float32 tensor of binary logits.

Your model will be evaluated on a held-out test set of 1,000 tiles from a geographic region not represented in the training data. The judge converts logits to predictions via sigmoid(logits) > 0.5 and computes the binary F1 score on the positive (Boma) class. F1 ≥ 0.75 is passing. The test set is not on the filesystem during training.

A smoke-test script /workspace/validate_model.py is provided. It calls load_model() and runs the returned module on a random (8, 10, 100, 100) float32 tensor, reporting only interface pass/fail, it does not measure performance. Submissions that fail to import, fail validation, or crash during evaluation receive an F1 score of 0.

Q4: Judge

The judge runs in a separate sandboxed VM with read-only access to the LLM's /workspace/ directory and exclusive access to /eval/test/ (the held-out test set). It executes the following steps in order. Any failure short-circuits to F1 = 0.

  1. Check that /workspace/model.py exists and is readable. → Fail if missing or unreadable.
  2. Import model in a subprocess with a 60-second timeout. → Fail on ImportError, SyntaxError, timeout, or any unhandled exception during import. Network calls during import also fail (sandbox blocks egress).
  3. Verify model.load_model exists, is callable, and accepts zero positional and zero required keyword arguments via inspect.signature. → Fail otherwise.
  4. Call load_model() with a 60-second timeout. → Fail if it raises, returns None, or returns a non-torch.nn.Module, or exceeds the timeout.
  5. Verify the returned module is in eval mode (module.training is False) and that all parameters reside on CPU. → Fail otherwise. (Prevents the LLM from leaving the model on a GPU device the judge VM doesn't have.)
  6. Run a forward pass on a random (8, 10, 100, 100) float32 tensor with reflectance-scale values. → Fail if forward raises, output shape ≠(8,), output dtype ≠ float32, or output contains NaN/±inf.
  7. Load the 1,000 held-out tiles from /eval/test/, shuffle with a fixed seed, batch at size 32, run forward in torch.no_grad() with a 5-minute total timeout. Convert logits to predictions via sigmoid(logits) > 0.5. → Fail on timeout, exception, or any non-finite output.

  8. Compute binary F1 on the positive (Boma) class using sklearn.metrics.f1_score(y_true, y_pred, pos_label=1, average='binary'). The test set is balanced 50/50 across positive and negative classes, computed once and held fixed across all evaluations.

  9. Return the continuous F1 score in [0.0, 1.0].

Passing threshold: F1 ≥ 0.75 is reported as the binary pass flag, but the continuous F1 is the actual reward signal. Continuous over binary because RL training benefits from gradient information, a model improving from 0.40 → 0.65 should receive credit even if it hasn't crossed the threshold yet, otherwise the entire region below 0.75 looks identical to the optimizer and learning collapses to a random walk until the threshold is breached.

Q5: Reward Hacking and Reward Denial

The judge measures F1 on a held-out test set. F1 is a proxy for the actual goal, a classifier that generalizes across East Africa. The gap between proxy and goal is the attack surface. Mitigations below close the major holes.

Test-set exfiltration. LLM finds /eval/test/ and trains on the labels. Mitigation: test set lives in a separate VM, mounted only at evaluation time. Filesystem reads outside the training directories blocked at the OS level.

Pretrained checkpoint download. My real Boma model is public, NeurIPS submission, GitHub repo, patent application. An LLM could download my own weights and submit them. Mitigation: no internet access in the sandbox. Backstop: judge hashes the submitted state_dict against known public checkpoints; matches get F1 = 0.

Constant predictions. Always predicting "Boma" inflates recall on imbalanced data. Mitigation: test set is balanced 50/50, capping constant predictions at F1 = 0.667, below the 0.75 threshold.

Metric gaming. Different F1 variants score the same predictions differently. Mitigation: the prompt specifies the exact sklearn call and threshold. No room for mismatched assumptions.

Judge crash exploitation. Returning NaN logits and hoping the judge mishandles them. Mitigation: every judge step explicitly checks for non-finite outputs and routes exceptions to F1 = 0. No fallback scoring logic exists to be exploited.

Iterative judge probing. The subtlest attack. If the LLM can call the judge during training, it'll overfit to judge idiosyncrasies, threshold behavior, batch-ordering quirks, producing a model tuned to the judge rather than the task. Mitigation: single-shot evaluation. The LLM has validate_model.py for interface checks only; it never returns a score and never touches the test set.

Reward denial, rigid interface checks. A genuinely good model fails on a technicality: wrong dtype, parameters left on GPU, forgotten .eval(). Mitigation: the prompt specifies the exact contract, and validate_model.py enables the LLM to self-check before submission. Cheaper to surface mismatches during training than at eval.

The combination keeps the genuine ML challenge intact, training a generalizing classifier from 200 labels, while closing the cheap paths to a high score.

Q6: Why I Designed This

I designed this environment because I've trained the exact model class it evaluates. In 2025, I built an end-to-end geospatial ML pipeline classifying indigenous Maasai settlements in Northern Tanzania: a TensorFlow CNN trained on 10,000 hand-labeled satellite tiles I created myself, the first dataset of its kind. The work won the NeurIPS 2024 Machine Learning for Social Impact track (1 of 4 winners from 330+ global submissions), continued as research at Harvard's Center for Geographic Analysis under Prof. Siqin Wang, and underpins US Patent Application 63/703,232. The deployed model directly informed the placement of 500 rainwater harvesting units serving 30,000+ Maasai across the Monduli District.

I work across three layers, and this environment touches all of them.

  1. Geospatial: I work on Google Earth Engine pipelines, Copernicus Sentinel-2 multispectral stacks, ArcGIS Pro, GeoPandas, and mAP/F1 evaluation on actual deployments, not just benchmark splits.
  2. Application engineering: I'm currently a backend engineer at the Statewide Database building voter-registration ingestion pipelines (Dagster, DuckDB, Pydantic), and I write Python, C++, and full-stack TypeScript daily.
  3. Machine learning: I can produce CNN design under label scarcity, semi-supervised retrieval on noisy archival data (the Cost of Colonialism project: OCR + NLP + embedding search over 17th-century Dutch East India Company shipping manifests), and the kind of failure-mode analysis the Q5 attack vectors require.

Alt environment I'd like to build: debug-a-broken-training-run, give the LLM a training script with 3–4 subtle bugs (wrong loss reduction, off-by-one LR scheduler, train/val split leak), judge held-out accuracy after the fix. Closer in spirit to SWE-Bench Pro but scoped to ML engineering.

%%

# Preference Model Take-Home Checklist

Hello!

Preference Model Take-Home Checklist

Time budget: 30 minutes. Format: PDF or Google Doc.

Pre-flight (2 min)

  • Skim Lilian Weng's reward hacking post — they linked it deliberately ✅ 2026-05-02
  • Open a doc, paste the 8 question prompts as headers so you don't miss any ✅ 2026-04-30

Q1 — Environment description (5 min)

  • State the task in one line (e.g., "few-shot Boma classification on multispectral satellite tiles") ✅ 2026-05-02
  • Explain why it's interesting — tie to real ML research workflow, not toy benchmarks ✅ 2026-05-02
  • Confirm it's hard for top LLMs (data scarcity + multispectral input ≠ ImageNet pretraining) ✅ 2026-05-02
  • Confirm prompt is unambiguous (file paths, tensor shapes, metric, threshold) ✅ 2026-05-02

Q2 — Tools, packages, data (3 min)

  • List packages: PyTorch, torchvision, rasterio (for .tif), numpy, sklearn (for F1) ✅ 2026-05-02
  • List compute: 1× GPU, time bound (e.g., 30 min wall-clock) ✅ 2026-05-02
  • List data: labeled train, unlabeled pool, hidden test (outside sandbox) ✅ 2026-05-02
  • State what's NOT available: internet, test set, pretrained Boma checkpoints ✅ 2026-05-02

Q3 — Write the prompt (5 min)

  • Model after the MNIST example exactly the same level of specificity ✅ 2026-05-02
  • Include: data paths, file formats, tensor shapes, output contract (load_model(), return type), metric, pass threshold, compute budget ✅ 2026-05-02
  • Read it back: could two engineers implement different things from it? If yes, tighten ✅ 2026-05-02

Q4 — Judge description (5 min)

  • Write as numbered steps (file exists → callable → returns nn.Module → shape check → run on test set → return score) ✅ 2026-05-02
  • State fail conditions explicitly at each step ✅ 2026-05-02
  • State success condition (F1 ≥ threshold, or continuous F1) ✅ 2026-05-02
  • Pick binary or continuous — justify in one line (continuous = better RL signal) ✅ 2026-05-02

Q5 — Reward hacking & denial (7 min — the differentiator)

  • List ≥ 4 specific hacking attack vectors with mitigations: ✅ 2026-05-02
    • Test set exfiltration → sandboxed filesystem
    • Pretrained model lookup → no internet / hash-check weights
    • Hardcoded/constant predictions → shuffle test inputs, smoke test with random tensor
    • Metric gaming on imbalanced data → spec exact F1 variant, balance test set
    • Judge crash exploits → validate finite outputs, exceptions = fail
  • List ≥ 1 reward denial risk (rigid interface checks → smoke test in prompt to surface mismatches early) ✅ 2026-05-02

Q6 — Why you designed this (2 min)

  • One paragraph: NeurIPS 2024 win, Harvard CGA research, US patent on Boma detection ✅ 2026-05-02
  • State the unfair advantage cleanly — you've trained this exact model class ✅ 2026-05-02
  • Optional: mention 1 alt environment (paper repro, training-run debug) as a one-liner ✅ 2026-05-02
  • github.com/roshantaneja ✅ 2026-05-02
  • Call out WaTER Vision repo specifically ✅ 2026-05-02

Q8 — Anything else (skip or 30 sec)

  • Skip if tight on time, OR one line on extension ideas (multi-task variants, harder geographic generalization splits) ✅ 2026-05-02

Final pass (1 min)

  • Voice check: no "I'm excited to," no "passionate about," no tricolons ✅ 2026-05-02
  • Length check: should fit on 2 pages, not 5 — they said 30 min for a reason ✅ 2026-05-02
  • Sharing settings: if Google Doc, set to "anyone with link can view" ✅ 2026-05-02

  • Send to Cai or Aleks per the instructions ✅ 2026-05-02

  • Skim Lilian Weng's reward hacking post — they linked it deliberately ✅ 2026-04-30

  • Open a doc, paste the 8 question prompts as headers so you don't miss any ✅ 2026-04-30
# Assignment

Preference Model Initial Assessment

This is our initial technical assessment. We are actively hiring candidates with strong backgrounds in AI/ML. Your assessment will be reviewed by one of our Machine Learning Engineers.

Objective

In this assessment, we ask you to design an RL environment for LLM training. An RL environment consists of a prompt, a judge, tools, and data. The prompt describes what the LLM should do, and the judge verifies that the LLM has done it correctly. The tools define what actions the LLM can take. The RL environment runs inside a virtual machine. For inspiration, you can take a look at SWE_Bench_Pro, which is a collection of realistic software engineering style problems. Unlike SWE-Bench, which is focused on software engineering, we are interested in AI/ML research and engineering.

Requirements

  1. Describe an environment you could build matching the requirements listed below. What makes this environment interesting? Note: we’re looking for your design proposal, not for you to write actual code.
    • The prompt should be something an AI/ML engineer or researcher might do.
    • The judge should give a high score when the LLM has solved the prompt successfully (true positives), and give a low score when it has not (true negatives).
    • The prompt and judge should be designed so as to allow for no false positives (reward hacking), and no false negatives (reward denial).
    • The judge should output either a binary score (pass/fail) or a continuous numerical score.
    • Environments should be challenging, even for top LLMs. If the LLM always receives a high score (or always passes), the environment is probably too easy.
    • The prompt should be unambiguous. Ambiguous prompt: “Train a model to classify digits." Unambiguous prompt: "Your task is to train a machine learning model that classifies MNIST digits with high accuracy. You have access to the MNIST training set in the folder /mnist/. When you are done, you should create a file /model.py which contains a function load_model(). load_model() should take no arguments, and return an MNIST classifier of type torch.nn.Module."
  2. The LLM has access to a command line tool which allows it to read, write, and run files, as well as GPUs/TPUs if necessary. What other tools, packages, and/or data will the LLM need in this environment?
  3. Write the prompt for this environment.
  4. Describe what the judge for this environment should do. What would cause the LLM to fail? What would cause the LLM to succeed?
    1. Example description: The judge performs the following steps: (1) Check that the file /model.py _exists. If not, fail. (2) Check that the load_model() function exists, and has the right arguments. It not, fail. (3) Import the model using the function load_model(). If exception, fail. (4) Finally, evaluate the accuracy of the model on the MNIST test set. Return a continuous numerical score based on the performance of the model on the test set. A higher accuracy leads to a better score, lower accuracy leads to lower score.
  5. Given your prompt and judge, is there any possibility of reward hacking or reward denial? If so, how? If not, why not?
  6. Why did you choose to design this environment? How does it relate to your experience/background? (Optional: describe other environments you might build.)
  7. Github Profile Link (optional: include links to repositories you want to highlight or projects you have built)
  8. Anything else you would like us to know?

%%