{"items":[{"description":"Implement a custom transformation class `MyTransform` to modify each input image tensor used by a PyTorch `DataLoader`. The DataLoader will apply your transform through a provided collate function, so your implementation must define how each image is transformed before training. The goal is to learn to integrate PyTorch transformations directly into data pipelines.","constraints":["Implement ONLY `MyTransform`.","You MUST return a tensor of the SAME shape and dtype.","The transform must NOT be the identity function.","It must be deterministic under a fixed random seed."],"id":"1","test_cases":[{"test":"import torch\nx = torch.rand(1,28,28)\nT = MyTransform()\ny = T(x)\nprint(y.shape == x.shape)","expected_output":"True"}],"approved_user_libraries":["torch"],"difficulty":"medium","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits (10 classes). Inputs initially scaled to [0,255] and shaped (N,1,28,28).","time_limits":{"test":"240 seconds","dev":"90 seconds"},"schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"X":"array([[[  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n           0.,   0.,   3.,  18.,  18.,  18., 126., 136., 175.,  26.,\n         166., 255., 247., 127.,   0.,   0.,   0.,   0.],..."},"output":{"y":"5"}},"category":"Data Pipelines","dataset":"MNIST","title":"MNIST: Pytorch DataLoader","model_specs":"Small CNN: Conv→ReLU→MaxPool x2, then Linear→ReLU→Linear(10). Batch size: 128, Optimizer: Adam lr=1e-3, epochs: 2.","memory_limit":"2 GB","createdAt":"2026-05-15T12:09:45.523000+00:00","time_limits_seconds":{"dev":90,"test":240},"type":"epoch","lower_is_better":false,"gpu":false},{"createdAt":"2026-05-13T21:01:39.924000+00:00","description":"Implement an attention mechanism for sentiment analysis on movie reviews. Your attention function will help a neural network learn which words in a review are most important for determining whether the sentiment is positive or negative.","constraints":["Your attention function must use only NumPy (no PyTorch, TensorFlow, or other ML frameworks)","Must return a NumPy array with correct shape","Must produce valid attention weights (non-negative, sum to 1 per query)","Must be deterministic (same input → same output)","Must handle batched inputs","Your function will be plugged into a PyTorch model for training (harness handles conversion)"],"id":"10","test_cases":[{"description":"Output shape matches expected","test":"import numpy as np\nQ = np.random.randn(4, 10, 32); K = np.random.randn(4, 10, 32); V = np.random.randn(4, 10, 32); out = attention(Q, K, V); print(out.shape == (4, 10, 32))","expected_output":"True"},{"description":"Works with different query and key lengths","test":"import numpy as np\nQ = np.random.randn(4, 5, 32); K = np.random.randn(4, 10, 32); V = np.random.randn(4, 10, 32); out = attention(Q, K, V); print(out.shape == (4, 5, 32))","expected_output":"True"},{"description":"Deterministic output","test":"import numpy as np\nnp.random.seed(42)\nQ = np.random.randn(2, 4, 16); K = np.random.randn(2, 4, 16); V = np.random.randn(2, 4, 16); out1 = attention(Q, K, V); out2 = attention(Q, K, V); print(np.allclose(out1, out2))","expected_output":"True"},{"description":"No NaN or Inf in output","test":"import numpy as np\nQ = np.random.randn(4, 8, 32); K = np.random.randn(4, 8, 32); V = np.random.randn(4, 8, 32); out = attention(Q, K, V); print(np.all(np.isfinite(out)))","expected_output":"True"}],"approved_user_libraries":["numpy"],"evaluation_metric":"Accuracy","input_format":{"K":"Key matrix of shape (batch_size, seq_len, dim)","Q":"Query matrix of shape (batch_size, seq_len, dim)","V":"Value matrix of shape (batch_size, seq_len, dim)"},"data_info":"IMDB Movie Reviews from TensorFlow's official .npz dataset: 50,000 reviews labeled as positive (1) or negative (0). Reviews are tokenized into sequences of word indices (vocabulary size 5,000). Sequences are padded/truncated to 80 tokens. Your attention mechanism helps the model focus on sentiment-bearing words.","difficulty":"medium","schema":"practical_question_v1","time_limits":{"test":"300 seconds","dev":"120 seconds"},"min_eval_metric":0.7,"output_format":{"output":"Attended output of shape (batch_size, seq_len, dim)"},"example":{"input":{"K":"np.array shape (64, 80, 64), key vectors","Q":"np.array shape (64, 80, 64), query vectors","V":"np.array shape (64, 80, 64), value vectors"},"output":{"attended":"np.array shape (64, 80, 64), weighted combination of values based on query-key compatibility"}},"category":"Attention & Transformers","title":"Design Your Own Attention Mechanism","model_specs":"PyTorch sequence classifier: Embedding(5000→64) → Q,K,V projection → [YOUR NUMPY ATTENTION] + Residual → GlobalAvgPool → Linear(64→1) → Sigmoid. Your NumPy attention function is wrapped in a custom PyTorch autograd Function for seamless integration.","topics":["attention","transformers","sequence modeling","deep learning","NLP"],"memory_limit":"2 GB","dataset":"IMDB Movie Reviews","function_signature":"attention(Q, K, V) -> output","time_limits_seconds":{"dev":120,"test":300},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a function to handle missing data (NaN values). Your function should fill in all NaN values with reasonable estimates. Your imputed data will be used to train a sklearn RandomForestRegressor on the Boston Housing dataset. Performance is evaluated on Mean Squared Error (MSE). Lower MSE is better. Since MSE is a loss metric, it is converted to an accuracy score using: val_accuracy = 1.0 - (MSE / 50.0). This means MSE of 0 gives accuracy of 1.0, MSE of 25 gives accuracy of 0.5, and the passing threshold of MSE < 35 corresponds to accuracy > 0.3.","type":"general","constraints":["Implement ONLY the `impute` function.","Output array must have the SAME shape as input X.","Output array must contain NO NaN or Inf values.","Only numpy is allowed."],"id":"11","test_cases":[{"test":"import numpy as np\nX = np.array([[1, np.nan], [3, 4], [np.nan, 6]])\nX_clean = impute(X)\nprint(X_clean.shape == X.shape)","expected_output":"True"},{"test":"import numpy as np\nX = np.array([[1, np.nan], [3, 4], [np.nan, 6]])\nX_clean = impute(X)\nprint(not np.isnan(X_clean).any())","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"medium","evaluation_metric":"MSE Accuracy","data_info":"Boston Housing dataset with 13 features predicting median home value ($1000s). ~400 training samples, ~100 test samples. ~15-25% values artificially set to NaN.","time_limits":{"test":"120 seconds","dev":"60 seconds"},"schema":"practical_question_v1","min_eval_metric":0.3,"example":{"input":{"X":"np.array with shape (n, 13) containing housing features with NaN values. The 13 features are: [0] CRIM (per capita crime rate), [1] ZN (proportion of residential land zoned for large lots), [2] INDUS (proportion of non-retail business acres), [3] CHAS (Charles River dummy: 1 if bounds river, 0 otherwise), [4] NOX (nitric oxide concentration), [5] RM (average rooms per dwelling), [6] AGE (proportion of old owner-occupied units), [7] DIS (distance to employment centers), [8] RAD (highway accessibility index), [9] TAX (property tax rate per $10k), [10] PTRATIO (pupil-teacher ratio), [11] B (proportion of Black residents), [12] LSTAT (% lower status population)"},"output":{"X_clean":"np.array with shape (n, 13) - same shape as X, no NaN values"}},"category":"Data Preprocessing","title":"Data Preprocessing: Handling Missing Values","model_specs":"RandomForestRegressor(n_estimators=100) trained on your cleaned data.","memory_limit":"1 GB","dataset":"Boston Housing","createdAt":"2026-05-15T12:10:20.607000+00:00","time_limits_seconds":{"dev":60,"test":120},"lower_is_better":false,"gpu":false},{"description":"Implement a function `train_step(model, x_batch, y_batch, lr)` that performs ONE step of gradient descent training. You must manually: (1) compute the forward pass, (2) compute the loss, (3) compute gradients using backpropagation, and (4) update the model parameters. Do NOT use torch.optim - update weights yourself using the gradients. This teaches how gradient descent actually works in PyTorch.","constraints":["Must compute forward pass through the model.","Must compute cross-entropy loss (or similar classification loss).","Must call backward() to compute gradients.","Must zero gradients before backward pass.","Must use torch.no_grad() when updating parameters.","Must return the loss value as a Python float.","Do not use Optim "],"id":"12","test_cases":[{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Linear(10, 2)\nx = torch.randn(4, 10)\ny = torch.tensor([0, 1, 0, 1])\nloss = train_step(model, x, y, lr=0.1)\nprint(isinstance(loss, float) and loss > 0)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 2))\nx = torch.randn(4, 10)\ny = torch.tensor([0, 1, 0, 1])\nloss = train_step(model, x, y, lr=0.1)\nprint(isinstance(loss, float))","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\n\n# Detect if torch.optim is used\n_optim_used = [False]\n_OrigSGD = torch.optim.SGD\n_OrigAdam = torch.optim.Adam\nclass _FakeSGD(_OrigSGD):\n    def __init__(self, *a, **kw):\n        _optim_used[0] = True\n        super().__init__(*a, **kw)\nclass _FakeAdam(_OrigAdam):\n    def __init__(self, *a, **kw):\n        _optim_used[0] = True\n        super().__init__(*a, **kw)\ntorch.optim.SGD = _FakeSGD\ntorch.optim.Adam = _FakeAdam\n\nmodel = nn.Linear(10, 2)\nx = torch.randn(4, 10)\ny = torch.tensor([0, 1, 0, 1])\ntrain_step(model, x, y, lr=0.1)\n\ntorch.optim.SGD = _OrigSGD\ntorch.optim.Adam = _OrigAdam\nprint(not _optim_used[0])","expected_output":"True"}],"difficulty":"medium","approved_user_libraries":["torch"],"evaluation_metric":"Accuracy","time_limits":{"test":"120 seconds","dev":"60 seconds"},"data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Trained with a 3-layer MLP (784->128->128->10).","schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"x_batch":"torch.Tensor of shape (batch_size, features)","model":"nn.Module (e.g., nn.Linear(784, 10) or nn.Sequential(...))","y_batch":"torch.Tensor of shape (batch_size,) with class labels","lr":"float, learning rate (e.g., 0.01)"},"output":{"loss":"float, the loss value for this batch"}},"category":"PyTorch Fundamentals","dataset":"MNIST","title":"PyTorch: Implement Your Own Gradient Descent Training Step","memory_limit":"2 GB","createdAt":"2026-05-15T12:10:24.590000+00:00","time_limits_seconds":{"dev":60,"test":120},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a function `train_model(model, X_train, y_train, X_val, y_val, epochs, batch_size, lr)` that trains a neural network using PyTorch. Use `torch.optim` for optimization and implement proper batching, shuffling, and validation. Return a list of dictionaries containing training metrics for each epoch.","constraints":["Must use torch.optim (SGD, Adam, or similar) for optimization.","Must implement mini-batch training with the specified batch_size.","Must shuffle training data each epoch.","Must compute validation accuracy and loss after each epoch.","Must return a list of dicts with keys: 'epoch', 'train_loss', 'val_loss', 'val_accuracy'.","Must use cross-entropy loss for classification."],"id":"13","test_cases":[{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Linear(10, 2)\nX_train = torch.randn(100, 10)\ny_train = torch.randint(0, 2, (100,))\nX_val = torch.randn(20, 10)\ny_val = torch.randint(0, 2, (20,))\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=2, batch_size=16, lr=0.01)\nprint(isinstance(history, list) and len(history) == 2)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Linear(10, 2)\nX_train = torch.randn(100, 10)\ny_train = torch.randint(0, 2, (100,))\nX_val = torch.randn(20, 10)\ny_val = torch.randint(0, 2, (20,))\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=2, batch_size=16, lr=0.01)\nprint(all('epoch' in h and 'train_loss' in h and 'val_loss' in h and 'val_accuracy' in h for h in history))","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Linear(10, 2)\nX_train = torch.randn(100, 10)\ny_train = torch.randint(0, 2, (100,))\nX_val = torch.randn(20, 10)\ny_val = torch.randint(0, 2, (20,))\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=3, batch_size=16, lr=0.01)\nprint(history[0]['epoch'] == 1 and history[-1]['epoch'] == 3)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 2))\nX_train = torch.randn(80, 10)\ny_train = torch.randint(0, 2, (80,))\nX_val = torch.randn(20, 10)\ny_val = torch.randint(0, 2, (20,))\nw_before = model[0].weight.clone().detach()\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=5, batch_size=16, lr=0.1)\nw_after = model[0].weight.clone().detach()\nprint(not torch.allclose(w_before, w_after))","expected_output":"True"}],"approved_user_libraries":["torch"],"difficulty":"easy","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Trained with a 3-layer MLP (784->128->128->10).","time_limits":{"test":"120 seconds","dev":"60 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"X_train":"torch.Tensor of shape (N, features)","batch_size":"int, mini-batch size","y_val":"torch.Tensor of shape (M,) with class labels","epochs":"int, number of training epochs","lr":"float, learning rate","X_val":"torch.Tensor of shape (M, features)","model":"nn.Module (e.g., nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10)))","y_train":"torch.Tensor of shape (N,) with class labels"},"output":{"history":"List[Dict] with keys 'epoch', 'train_loss', 'val_accuracy'"}},"category":"PyTorch Fundamentals","title":"PyTorch: Build a Complete Training Loop","dataset":"MNIST","memory_limit":"2 GB","createdAt":"2026-05-15T12:10:28.906000+00:00","time_limits_seconds":{"dev":60,"test":120},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a dimensionality reduction class `MyReducer` that projects high-dimensional data down to 10 dimensions using NumPy. Your reducer will be evaluated by training a k-NN classifier on the reduced MNIST data and measuring accuracy.","type":"general","constraints":["Implement using NumPy only. No sklearn, scipy.linalg, or other high-level libraries.","Must implement: fit(X), transform(X), and fit_transform(X).","transform() must return shape (n_samples, 10).","The reduction must be deterministic (same input -> same output).","Cannot simply select features or random subsets.","Projection matrix must be at most shape (n_features, 10)."],"id":"14","test_cases":[{"test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(100, 50)\nreducer = MyReducer()\nreducer.fit(X)\nX_t = reducer.transform(X)\nprint(X_t.shape == (100, 10))","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(100, 50)\nreducer = MyReducer()\nX_t1 = reducer.fit_transform(X)\nX_t2 = reducer.transform(X)\nprint(np.allclose(X_t1, X_t2))","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(80, 30)\nreducer1 = MyReducer()\nreducer2 = MyReducer()\nX_t1 = reducer1.fit_transform(X)\nX_t2 = reducer2.fit_transform(X)\nprint(np.allclose(X_t1, X_t2))","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","data_info":"Validation: MNIST (2000 train, 500 test, 784 -> 10 dims). Test: MNIST (5000 train, 1000 test, 784 -> 10 dims). Evaluated with k-NN classifier.","time_limits":{"test":"180 seconds","dev":"90 seconds"},"schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"X":"np.array of shape (n_samples, n_features), e.g. shape (5000, 784) for 5000 MNIST images"},"output":{"X_reduced":"np.array of shape (n_samples, n_components), e.g. shape (5000, 50) for 50-dimensional projection"}},"category":"Dimensionality Reduction","title":"Numpy: Design Your Own Dimensionality Reduction","dataset":"MNIST","memory_limit":"2 GB","model_specifications":"sklearn KNeighborsClassifier (k=5) trained on MNIST reduced to 10 dimensions.","model_specs":"sklearn KNeighborsClassifier (k=5) trained on MNIST reduced to 10 dimensions.","createdAt":"2026-05-15T12:10:51.564000+00:00","time_limits_seconds":{"dev":90,"test":180},"lower_is_better":false,"gpu":false},{"description":"Implement a dimensionality reduction class `MyReducer` that projects high-dimensional data down to 10 dimensions. You can use sklearn and NumPy. Your reducer will be evaluated by training a k-NN classifier on the reduced MNIST data and measuring accuracy.","constraints":["Must implement: fit(X), transform(X), and fit_transform(X).","transform() must return shape (n_samples, 10).","The reduction must be deterministic (same input -> same output).","Can use sklearn and numpy only.","Cannot simply select features or random subsets."],"id":"15","test_cases":[{"test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(100, 50)\nreducer = MyReducer()\nreducer.fit(X)\nX_t = reducer.transform(X)\nprint(X_t.shape == (100, 10))","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(100, 50)\nreducer = MyReducer()\nX_t1 = reducer.fit_transform(X)\nX_t2 = reducer.transform(X)\nprint(np.allclose(X_t1, X_t2))","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(80, 30)\nreducer1 = MyReducer()\nreducer2 = MyReducer()\nX_t1 = reducer1.fit_transform(X)\nX_t2 = reducer2.fit_transform(X)\nprint(np.allclose(X_t1, X_t2))","expected_output":"True"}],"approved_user_libraries":["numpy","sklearn"],"difficulty":"easy","evaluation_metric":"Accuracy","data_info":"Validation: MNIST (2000 train, 500 test, 784 -> 10 dims). Test: MNIST (5000 train, 1000 test, 784 -> 10 dims). Evaluated with k-NN classifier.","time_limits":{"test":"180 seconds","dev":"90 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"X":"np.array of shape (n_samples, n_features), e.g. shape (5000, 784) for 5000 MNIST images"},"output":{"X_reduced":"np.array of shape (n_samples, 10)"}},"category":"Dimensionality Reduction","title":"Dimensionality Reduction with Sklearn","model_specs":"sklearn KNeighborsClassifier (k=5) trained on MNIST reduced to 10 dimensions.","memory_limit":"2 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:40.633000+00:00","time_limits_seconds":{"dev":90,"test":180},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"You are training a digit classifier on Colored MNIST, where each digit is colored with a class-correlated color (e.g., 0s are usually red, 1s are usually green). During training, colors are 80% correlated with digit class, creating a shortcut the model can exploit. At test time, colors are only 30% correlated, so a model relying on color will perform poorly. Your task is to implement a `FeatureDeconfounder` layer that removes the linear influence of color metadata from neural network features during training, forcing the model to learn digit shapes instead of color shortcuts.","constraints":["Implement ONLY the FeatureDeconfounder class with fit() and transform() methods.","fit() must precompute Sigma_inv = (X^T X + reg*I)^{-1} from training metadata.","transform() must compute beta = Sigma_inv @ X^T @ f and return residual = f - X @ beta.","transform() must allow gradients to flow through for end-to-end training.","Must handle both numpy arrays and PyTorch tensors as input."],"id":"16","test_cases":[{"test":"import torch\ndeconf = FeatureDeconfounder()\nmeta = torch.rand(100, 3)\ndeconf.fit(meta)\nf = torch.rand(10, 64)\nout = deconf.transform(f, torch.rand(10, 3))\nprint(out.shape == f.shape)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\ndeconf = FeatureDeconfounder()\nmeta = torch.rand(100, 3)\ndeconf.fit(meta)\nf = torch.rand(10, 64, requires_grad=True)\nm = torch.rand(10, 3)\nout = deconf.transform(f, m)\nloss = out.sum()\nloss.backward()\nprint(f.grad is not None)","expected_output":"True"}],"approved_user_libraries":["torch","numpy"],"evaluation_metric":"Accuracy","data_info":"Colored MNIST dataset where each grayscale digit is colorized. Training data has 80% color-class correlation (digit 0 is usually red, digit 1 is usually green, etc.). Test data has 30% correlation (reduced bias). Metadata is 3-dimensional RGB color values. The model must learn to classify digits by shape, not color, to succeed on the less-biased test set.","time_limits":{"test":180,"validation":120},"difficulty":"hard","schema":"FeatureDeconfounder class with fit(metadata) and transform(features, metadata) methods","min_eval_metric":0.5,"example":{"input":{"features":"torch.Tensor of shape (M, D) - batch of encoder features","colors_train":"torch.Tensor of shape (N, 3) - all training colors for fit()","metadata":"torch.Tensor of shape (M, K) - corresponding color values (RGB)"},"output":{"note":"Gradients flow through transform() for end-to-end training","residual":"torch.Tensor of shape (M, D) - features with color influence removed"}},"category":"Feature Deconfounding","title":"Feature Deconfounder for Biased Image Data","model_specs":"Small CNN: Conv(3,16)→ReLU→Pool→Conv(16,32)→ReLU→Pool→FC(1568,64). Training: Adam lr=1e-3, batch_size=128, epochs=5.","memory_limit":"4GB","dataset":"Colored MNIST","createdAt":"2026-05-15T12:11:09.647000+00:00","time_limits_seconds":{"dev":60,"test":180},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a function `train_model(model, X_train, y_train, X_val, y_val, epochs, batch_size, lr)` that trains a neural network using PyTorch. Use `torch.optim` for optimization and implement proper batching, shuffling, and validation. Return a list of dictionaries containing training metrics for each epoch.","constraints":["Must use torch.optim (SGD, Adam, or similar) for optimization.","Must implement mini-batch training with the specified batch_size.","Must shuffle training data each epoch.","Must compute validation accuracy and loss after each epoch.","Must return a list of dicts with keys: 'epoch', 'train_loss', 'val_loss', 'val_accuracy'.","Must use cross-entropy loss for classification."],"id":"17","test_cases":[{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Linear(10, 2)\nX_train = torch.randn(100, 10)\ny_train = torch.randint(0, 2, (100,))\nX_val = torch.randn(20, 10)\ny_val = torch.randint(0, 2, (20,))\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=2, batch_size=16, lr=0.01)\nprint(isinstance(history, list) and len(history) == 2)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Linear(10, 2)\nX_train = torch.randn(100, 10)\ny_train = torch.randint(0, 2, (100,))\nX_val = torch.randn(20, 10)\ny_val = torch.randint(0, 2, (20,))\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=2, batch_size=16, lr=0.01)\nprint(all('epoch' in h and 'train_loss' in h and 'val_loss' in h and 'val_accuracy' in h for h in history))","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Linear(10, 2)\nX_train = torch.randn(100, 10)\ny_train = torch.randint(0, 2, (100,))\nX_val = torch.randn(20, 10)\ny_val = torch.randint(0, 2, (20,))\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=3, batch_size=16, lr=0.01)\nprint(history[0]['epoch'] == 1 and history[-1]['epoch'] == 3)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\nmodel = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 2))\nX_train = torch.randn(80, 10)\ny_train = torch.randint(0, 2, (80,))\nX_val = torch.randn(20, 10)\ny_val = torch.randint(0, 2, (20,))\nw_before = model[0].weight.clone().detach()\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=5, batch_size=16, lr=0.1)\nw_after = model[0].weight.clone().detach()\nprint(not torch.allclose(w_before, w_after))","expected_output":"True"}],"approved_user_libraries":["torch"],"difficulty":"easy","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Trained with a 3-layer MLP (784->128->128->10).","time_limits":{"test":"120 seconds","dev":"60 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"X_train":"torch.Tensor of shape (N, features)","batch_size":"int, mini-batch size","y_val":"torch.Tensor of shape (M,) with class labels","epochs":"int, number of training epochs","lr":"float, learning rate","X_val":"torch.Tensor of shape (M, features)","model":"nn.Module (e.g., nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10)))","y_train":"torch.Tensor of shape (N,) with class labels"},"output":{"history":"List[Dict] with keys 'epoch', 'train_loss', 'val_accuracy'"}},"category":"PyTorch Fundamentals","title":"PyTorch: Build a Complete Training Loop","dataset":"MNIST","memory_limit":"2 GB","createdAt":"2026-05-15T12:11:15.521000+00:00","time_limits_seconds":{"dev":60,"test":120},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a `train` function that takes standardized training data and randomly initialized weights, and returns trained weights for a linear regression model.\n\nThe harness handles everything else: data loading, feature standardization, weight initialization, prediction (`y = X @ W + b`), and evaluation. You just define **how the weights get updated**.\n\nYou may use ANY training strategy you like: gradient descent, the normal equation, momentum, adaptive learning rates, mini-batching, or anything else -- as long as you only use NumPy and the returned `(W, b)` produce good predictions.","type":"general","constraints":["Use ONLY numpy - no sklearn, PyTorch, TensorFlow, or other ML libraries","Your function signature must be: train(X, y, W, b) -> (W, b)","X and y are already standardized (zero mean, unit variance) -- do NOT re-standardize","W is a 1-D array of shape (n_features,), b is a float","Must return W as a 1-D numpy array and b as a float (or 0-d array)","Must be deterministic under a fixed numpy random seed","Any training strategy is valid!"],"id":"18","test_cases":[{"description":"Returns correct types and shapes","test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(100, 5)\ny = np.random.randn(100)\nW = np.random.randn(5) * 0.01\nb = 0.0\nW_out, b_out = train(X, y, W, b)\nprint(W_out.shape == (5,) and np.isscalar(b_out) or b_out.ndim == 0)","expected_output":"True"},{"description":"Output contains no NaN or Inf","test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(100, 5)\ny = np.random.randn(100)\nW = np.random.randn(5) * 0.01\nb = 0.0\nW_out, b_out = train(X, y, W, b)\nprint(np.all(np.isfinite(W_out)) and np.isfinite(b_out))","expected_output":"True"},{"description":"Actually learns (predictions improve over random weights)","test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(200, 3)\ntrue_w = np.array([1.0, -2.0, 0.5])\ny = X @ true_w + 0.1 * np.random.randn(200)\nW = np.random.randn(3) * 0.01\nb = 0.0\nW_out, b_out = train(X, y, W, b)\npreds_before = X @ W + b\npreds_after = X @ W_out + b_out\nmse_before = np.mean((y - preds_before)**2)\nmse_after = np.mean((y - preds_after)**2)\nprint(mse_after < mse_before * 0.1)","expected_output":"True"},{"description":"Deterministic output","test":"import numpy as np\nnp.random.seed(42)\nX = np.random.randn(100, 4)\ny = np.random.randn(100)\nW = np.random.randn(4) * 0.01\nb = 0.0\nnp.random.seed(99)\nW1, b1 = train(X.copy(), y.copy(), W.copy(), b)\nnp.random.seed(99)\nW2, b2 = train(X.copy(), y.copy(), W.copy(), b)\nprint(np.allclose(W1, W2) and np.isclose(b1, b2))","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"easy","evaluation_metric":"R2","data_info":"California Housing dataset from sklearn: 20,640 samples with 8 numerical features (MedInc, HouseAge, AveRooms, AveBedrms, Population, AveOccup, Latitude, Longitude). Target is median house value in $100,000s. Both features and target are standardized (zero mean, unit variance) before being passed to your function.","time_limits":{"test":"60 seconds","dev":"30 seconds"},"schema":"practical_question_v1","min_eval_metric":0.5,"example":{"input":{"X":"numpy array of shape (3000, 8) -- standardized training features","y":"numpy array of shape (3000,) -- standardized target values","W":"numpy array of shape (8,) -- randomly initialized weights","b":"float -- initialized bias (0.0)"},"output":{"W":"numpy array of shape (8,) -- trained weights","b":"float -- trained bias"}},"category":"Regression","title":"Train a Linear Regression Model","model_specs":"Linear model: y = X @ W + b. Harness standardizes data, initializes W ~ N(0, 0.01), b = 0. User trains weights, harness evaluates R^2 on held-out data.","memory_limit":"1 GB","dataset":"California Housing","createdAt":"2026-05-15T12:11:17.612000+00:00","time_limits_seconds":{"dev":30,"test":60},"lower_is_better":false,"gpu":false},{"description":"Implement a `train_tokenizer` function that learns a vocabulary from a training corpus of names and returns `encode` and `decode` functions.\n\nThe harness will use your tokenizer to train a bigram language model on the tokenized corpus, then evaluate how well that model predicts held-out names. A better tokenizer creates more predictable token sequences, leading to lower bits-per-character (BPC) on held-out data.\n\nYou may use ANY tokenization strategy: character-level, byte pair encoding (BPE), frequency-based subword merging, common n-gram extraction, word-level with character fallback, or anything creative -- as long as you only use standard Python (no external libraries).","type":"general","constraints":["Use ONLY the Python standard library -- no numpy, sklearn, or other external libraries","Your function signature must be: train_tokenizer(corpus, vocab_size) -> (encode, decode)","corpus is a list[str] of training documents (names), vocab_size is an int (256)","encode: str -> list[int], all token IDs must be in range [0, vocab_size)","decode: list[int] -> str, must satisfy decode(encode(text)) == text for all text","Must handle any text composed of characters seen in the training corpus","Must be deterministic -- same input always produces same output"],"id":"19","test_cases":[{"description":"Returns callable encode and decode","test":"corpus = ['hello', 'world', 'help', 'held', 'helm', 'hero']\nencode, decode = train_tokenizer(corpus, 64)\nprint(callable(encode) and callable(decode))","expected_output":"True"},{"description":"encode returns list of ints in range","test":"corpus = ['hello', 'world', 'help', 'held', 'helm', 'hero']\nencode, decode = train_tokenizer(corpus, 64)\ntokens = encode('hello')\nprint(isinstance(tokens, list) and all(isinstance(t, int) and 0 <= t < 64 for t in tokens))","expected_output":"True"},{"description":"Lossless roundtrip on training data","test":"corpus = ['hello', 'world', 'help', 'held', 'helm', 'hero']\nencode, decode = train_tokenizer(corpus, 64)\nprint(all(decode(encode(w)) == w for w in corpus))","expected_output":"True"},{"description":"Handles unseen text with known characters","test":"corpus = ['hello', 'world', 'help', 'held', 'helm', 'hero']\nencode, decode = train_tokenizer(corpus, 64)\nprint(decode(encode('herd')) == 'herd' and decode(encode('howl')) == 'howl')","expected_output":"True"}],"difficulty":"medium","evaluation_metric":"BPC_improvement","data_info":"Names dataset (32,033 human names) from Karpathy's makemore project. Each name is a short lowercase string (average ~6 characters, alphabet of 26 letters). Training set: 25,000 names. The harness trains a bigram language model on the tokenized training data and evaluates bits-per-character on held-out names.","time_limits":{"test":"120 seconds","dev":"120 seconds"},"approved_user_libraries":["math"],"schema":"practical_question_v1","min_eval_metric":-0.05,"example":{"input":{"corpus":"list[str] -- 25,000 training names, e.g. ['emma', 'olivia', 'ava', ...]","vocab_size":"int -- maximum vocabulary size (256)"},"output":{"encode":"callable: str -> list[int], e.g. encode('emma') -> [4, 26, 0] (token IDs in [0, 256))","decode":"callable: list[int] -> str, e.g. decode([4, 26, 0]) -> 'emma'"}},"category":"NLP","title":"Build a Tokenizer for Language Modeling","model_specs":"Harness loads 32K names, gives 25K to user's train_tokenizer(). Then trains a bigram language model on the tokenized training data with interpolation smoothing. Evaluates bits-per-character (BPC) on held-out names. Score = char_baseline_BPC - user_BPC. Character-level scores ~0.0 (baseline). Better tokenizers score higher.","memory_limit":"1 GB","dataset":"Names (Karpathy makemore)","createdAt":"2026-05-13T21:01:41.183000+00:00","time_limits_seconds":{"dev":120,"test":120},"lower_is_better":false,"gpu":false},{"description":"Implement a function `build_model()` that returns a PyTorch `nn.Module` to classify MNIST digits. You may use ANY layers/ops, but if your model has more than **2048 trainable parameters**, your reported accuracy will be **0**. Keep it tiny and clever (e.g., global pooling, grouped/depthwise convs, shared weights, etc.).","constraints":["Implement ONLY `build_model()`; return an `nn.Module` instance.","All trainable parameters (requires_grad=True) count toward the 2048 limit.","If param_count > 2048, the grader sets accuracy to 0.","Use GPU if available."],"id":"2","test_cases":[{"test":"import torch, torch.nn as nn\nm = build_model()\nprint(isinstance(m, nn.Module))","expected_output":"True"},{"test":"import torch, torch.nn as nn\nm = build_model()\nparams = sum(p.numel() for p in m.parameters() if p.requires_grad)\nprint(params <= 2048)","expected_output":"True"}],"approved_user_libraries":["torch","numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits (10 classes). Inputs scaled to [0,1] and shaped (N,1,28,28).","time_limits":{"test":"240 seconds","dev":"90 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"X":"array([[[  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.],\n        [  0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,\n         0.,   0.,   3.,  18.,  18.,  18., 126., 136., 175.,  26.,\n       166., 255., 247., 127.,   0.,   0.,   0.,   0.], ... ]])  # shape (1,28,28), values in [0,255] before scaling","y":"5"},"output":{"note":"If param_count > 2048 → accuracy is set to 0 in the final JSON.","param_count":"≤ 2048"}},"category":"Model Architecture","title":"MNIST: Design-Your-Own tiny Pytorch Model","model_specs":"You choose the architecture. Training: Adam lr=1e-3, batch size=128, epochs=10.","memory_limit":"2 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:41.403000+00:00","time_limits_seconds":{"dev":90,"test":240},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a `train` function that learns to classify handwritten digits using ONLY Python and the math standard library.\n\nThe harness loads the sklearn digits dataset (8x8 pixel images of digits 0-9, giving 64 features per sample), normalizes pixel values to [0, 1], and passes training and validation data as nested Python lists.\n\nYou must return a `predict` function that classifies new images. You may use ANY approach: nearest-neighbor, centroid classifier, logistic regression, neural network with hand-rolled backprop, decision trees, or anything creative -- as long as you only use Python and the math module.","type":"general","constraints":["Use ONLY Python and the math standard library -- no numpy, sklearn, torch, or other libraries","Your function signature must be: train(X_train, y_train, X_val, y_val, n_classes) -> predict","X_train and X_val are list[list[float]], each inner list has 64 floats in [0, 1]","y_train and y_val are list[int], each value in [0, n_classes)","predict must be callable: list[list[float]] -> list[int]","predict must return a list of integer class predictions, one per input sample","Must complete training and prediction within the time limit"],"id":"20","test_cases":[{"description":"Returns callable predict function","test":"X = [[0.1]*64 for _ in range(20)]\ny = [i % 10 for i in range(20)]\npredict = train(X, y, X[:5], y[:5], 10)\nprint(callable(predict))","expected_output":"True"},{"description":"predict returns list of ints","test":"X = [[0.1]*64 for _ in range(20)]\ny = [i % 10 for i in range(20)]\npredict = train(X, y, X[:5], y[:5], 10)\npreds = predict(X[:5])\nprint(isinstance(preds, list) and all(isinstance(p, int) for p in preds))","expected_output":"True"},{"description":"predict returns correct number of predictions","test":"X = [[0.1]*64 for _ in range(20)]\ny = [i % 10 for i in range(20)]\npredict = train(X, y, X[:5], y[:5], 10)\npreds = predict(X[:8])\nprint(len(preds) == 8)","expected_output":"True"},{"description":"Predictions are valid class indices","test":"X = [[0.1]*64 for _ in range(20)]\ny = [i % 10 for i in range(20)]\npredict = train(X, y, X[:5], y[:5], 10)\npreds = predict(X)\nprint(all(0 <= p < 10 for p in preds))","expected_output":"True"}],"approved_user_libraries":["math","random"],"difficulty":"hard","evaluation_metric":"Accuracy","data_info":"sklearn digits dataset: 1,797 samples of 8x8 pixel grayscale images of handwritten digits (0-9). Each sample has 64 features (pixel intensities normalized to [0, 1]). 10 classes. Training: 1,000 samples, validation: 400 samples, test: 500 samples. Data is passed as nested Python lists.","time_limits":{"test":"60 seconds","dev":"60 seconds"},"schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"X_train":"list[list[float]] -- 1000 training images, each 64 floats in [0, 1]","n_classes":"int -- number of classes (10)","y_train":"list[int] -- 1000 training labels in [0, 9]","X_val":"list[list[float]] -- 400 validation images","y_val":"list[int] -- 400 validation labels"},"output":{"predict":"callable: list[list[float]] -> list[int], returns predicted class for each input image"}},"category":"Classification","title":"Build a Digit Classifier from Scratch","model_specs":"Harness loads sklearn digits (8x8 images, 64 features, 10 classes), normalizes to [0,1], converts to nested Python lists. User implements train() returning predict(). Evaluated on accuracy. User may only use Python + math module.","dataset":"Digits (sklearn)","memory_limit":"2 GB","createdAt":"2026-05-13T21:01:41.516000+00:00","time_limits_seconds":{"dev":60,"test":60},"lower_is_better":false,"gpu":false},{"description":"You are given a regression dataset that has been deliberately over-engineered with polynomial features and noise columns. A naive approach (e.g., ordinary least squares or unregularized gradient descent) will badly overfit: it achieves near-perfect training R² but catastrophically negative test R².\n\nYour task: implement a `train` function that returns a `predict` callable which actually generalizes to unseen data. You must use ONLY NumPy.\n\nThe harness generates degree-3 polynomial features plus 100 pure noise columns from the original 8-feature California Housing dataset, giving ~264 features but only ~250 training samples. This makes the problem deliberately overparameterized. Unregularized OLS will get Train R² ≈ 1.0 but Val R² ≈ -5.0 (worse than guessing the mean!).","type":"general","constraints":["Use ONLY numpy - no sklearn, PyTorch, TensorFlow, or other ML libraries","Your function signature must be: train(X_train, y_train, X_val, y_val) -> predict","predict must be callable: predict(X) -> y_pred (numpy array)","X_train and X_val are already standardized -- do NOT re-standardize","y_train and y_val are raw target values (not standardized)","Must be deterministic under a fixed numpy random seed","Any strategy is valid: L1, L2, ElasticNet, feature selection, early stopping, etc."],"id":"21","test_cases":[{"description":"Returns a callable predict function","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 50)\ny_tr = np.random.randn(100)\nX_v = np.random.randn(20, 50)\ny_v = np.random.randn(20)\npredict = train(X_tr, y_tr, X_v, y_v)\nprint(callable(predict))","expected_output":"True"},{"description":"predict returns correct shape","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 50)\ny_tr = np.random.randn(100)\nX_v = np.random.randn(20, 50)\ny_v = np.random.randn(20)\npredict = train(X_tr, y_tr, X_v, y_v)\npreds = predict(X_v)\nprint(preds.shape == (20,))","expected_output":"True"},{"description":"predict output contains no NaN or Inf","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 50)\ny_tr = np.random.randn(100)\nX_v = np.random.randn(20, 50)\ny_v = np.random.randn(20)\npredict = train(X_tr, y_tr, X_v, y_v)\npreds = predict(X_v)\nprint(np.all(np.isfinite(preds)))","expected_output":"True"},{"description":"Deterministic output","test":"import numpy as np\nX_tr = np.random.randn(80, 40)\ny_tr = np.random.randn(80)\nX_v = np.random.randn(20, 40)\ny_v = np.random.randn(20)\nnp.random.seed(99)\np1 = train(X_tr.copy(), y_tr.copy(), X_v.copy(), y_v.copy())\npreds1 = p1(X_v.copy())\nnp.random.seed(99)\np2 = train(X_tr.copy(), y_tr.copy(), X_v.copy(), y_v.copy())\npreds2 = p2(X_v.copy())\nprint(np.allclose(preds1, preds2))","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"medium","evaluation_metric":"R2","data_info":"California Housing dataset expanded with degree-3 polynomial features (~164 from 8 original) plus 100 pure noise columns = ~264 total features. Only 250 training samples. The data is deliberately overparameterized (more features than samples) so that unregularized regression perfectly memorizes training data (R² ≈ 1.0) but fails catastrophically on validation (R² ≈ -5.0). Features are standardized to zero mean, unit variance. Target (median house value in $100Ks) is NOT standardized.","time_limits":{"test":"60 seconds","dev":"30 seconds"},"schema":"practical_question_v1","min_eval_metric":0.45,"example":{"input":{"X_train":"numpy array of shape (250, ~264) -- standardized polynomial + noise features (more features than samples!)","y_val":"numpy array of shape (300,) -- validation target values","y_train":"numpy array of shape (250,) -- target values (median house value in $100Ks)","X_val":"numpy array of shape (300, ~264) -- standardized validation features"},"output":{"predict":"callable: predict(X) -> numpy array of shape (n_samples,)"}},"category":"Regularized Regression","title":"Fix Overfitting with Regularization (NumPy)","model_specs":"Linear model. Harness creates ~264 features (degree-3 polynomial + 100 noise) from 8 original California Housing features with only 250 training samples. More features than samples = guaranteed overfitting without regularization. User must implement regularization to generalize. Evaluated on R² on held-out data.","memory_limit":"1 GB","dataset":"California Housing (polynomial expanded + noise)","createdAt":"2026-05-13T21:01:41.684000+00:00","time_limits_seconds":{"dev":30,"test":60},"lower_is_better":false,"gpu":false},{"description":"You are given a regression dataset that has been deliberately over-engineered with polynomial features and noise columns. A naive approach (e.g., `LinearRegression()`) will badly overfit: near-perfect training R² but catastrophically negative test R².\n\nYour task: implement a `train` function that returns a `predict` callable which actually generalizes to unseen data. You can use sklearn and NumPy.\n\nThe harness generates degree-3 polynomial features plus 100 pure noise columns from the original 8-feature California Housing dataset, giving ~264 features but only ~250 training samples. `LinearRegression()` will get Train R² ≈ 1.0 but Val R² ≈ -5.0. You need sklearn's regularized models to fix this.","type":"general","constraints":["You can use numpy and sklearn","Your function signature must be: train(X_train, y_train, X_val, y_val) -> predict","predict must be callable: predict(X) -> y_pred (numpy array)","X_train and X_val are already standardized -- do NOT re-standardize","y_train and y_val are raw target values (not standardized)","Must be deterministic under a fixed numpy random seed","Any strategy is valid: Ridge, Lasso, ElasticNet, feature selection, Pipeline, etc."],"id":"22","test_cases":[{"description":"Returns a callable predict function","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 50)\ny_tr = np.random.randn(100)\nX_v = np.random.randn(20, 50)\ny_v = np.random.randn(20)\npredict = train(X_tr, y_tr, X_v, y_v)\nprint(callable(predict))","expected_output":"True"},{"description":"predict returns correct shape","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 50)\ny_tr = np.random.randn(100)\nX_v = np.random.randn(20, 50)\ny_v = np.random.randn(20)\npredict = train(X_tr, y_tr, X_v, y_v)\npreds = predict(X_v)\nprint(preds.shape == (20,))","expected_output":"True"},{"description":"predict output contains no NaN or Inf","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 50)\ny_tr = np.random.randn(100)\nX_v = np.random.randn(20, 50)\ny_v = np.random.randn(20)\npredict = train(X_tr, y_tr, X_v, y_v)\npreds = predict(X_v)\nprint(np.all(np.isfinite(preds)))","expected_output":"True"},{"description":"Deterministic output","test":"import numpy as np\nX_tr = np.random.randn(80, 40)\ny_tr = np.random.randn(80)\nX_v = np.random.randn(20, 40)\ny_v = np.random.randn(20)\nnp.random.seed(99)\np1 = train(X_tr.copy(), y_tr.copy(), X_v.copy(), y_v.copy())\npreds1 = p1(X_v.copy())\nnp.random.seed(99)\np2 = train(X_tr.copy(), y_tr.copy(), X_v.copy(), y_v.copy())\npreds2 = p2(X_v.copy())\nprint(np.allclose(preds1, preds2))","expected_output":"True"}],"approved_user_libraries":["numpy","sklearn"],"difficulty":"easy","evaluation_metric":"R2","data_info":"California Housing dataset expanded with degree-3 polynomial features (~164 from 8 original) plus 100 pure noise columns = ~264 total features. Only 250 training samples. The data is deliberately overparameterized (more features than samples) so that unregularized LinearRegression() perfectly memorizes training data (R² ≈ 1.0) but fails catastrophically on validation (R² ≈ -5.0). Features are standardized to zero mean, unit variance. Target (median house value in $100Ks) is NOT standardized.","time_limits":{"test":"60 seconds","dev":"30 seconds"},"schema":"practical_question_v1","min_eval_metric":0.5,"example":{"input":{"X_train":"numpy array of shape (250, ~264) -- standardized polynomial + noise features (more features than samples!)","y_val":"numpy array of shape (300,) -- validation target values","y_train":"numpy array of shape (250,) -- target values (median house value in $100Ks)","X_val":"numpy array of shape (300, ~264) -- standardized validation features"},"output":{"predict":"callable: predict(X) -> numpy array of shape (n_samples,)"}},"category":"Regularized Regression","title":"Fix Overfitting with Regularization (Sklearn)","model_specs":"Sklearn linear model. Harness creates ~264 features (degree-3 polynomial + 100 noise) from 8 original California Housing features with only 250 training samples. More features than samples = guaranteed overfitting without regularization. User must use regularized sklearn models to generalize. Evaluated on R² on held-out data.","memory_limit":"1 GB","dataset":"California Housing (polynomial expanded + noise)","createdAt":"2026-05-13T21:01:41.782000+00:00","time_limits_seconds":{"dev":30,"test":60},"lower_is_better":false,"gpu":false},{"description":"You are given a binary classification dataset where the target is either 0 or 1. Implement a `train` function that returns a `predict` callable which classifies new samples accurately.\n\nThe harness loads the Breast Cancer dataset from sklearn (30 numerical features describing cell nuclei measurements), standardizes the features, splits into train/val/test, and calls your function. Your `predict` must return an array of 0s and 1s.\n\nYou can use any sklearn model or approach you like.","type":"general","constraints":["You can use numpy and sklearn","Your function signature must be: train(X_train, y_train, X_val, y_val) -> predict","predict must be callable: predict(X) -> y_pred (numpy array of 0s and 1s)","X_train and X_val are already standardized -- do NOT re-standardize","y_train and y_val are integer arrays with values 0 or 1","Must be deterministic under a fixed numpy random seed","Any sklearn model or strategy is valid"],"id":"23","test_cases":[{"description":"Returns a callable predict function","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 10)\ny_tr = np.random.randint(0, 2, 100)\nX_v = np.random.randn(20, 10)\ny_v = np.random.randint(0, 2, 20)\npredict = train(X_tr, y_tr, X_v, y_v)\nprint(callable(predict))","expected_output":"True"},{"description":"predict returns correct shape","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 10)\ny_tr = np.random.randint(0, 2, 100)\nX_v = np.random.randn(20, 10)\ny_v = np.random.randint(0, 2, 20)\npredict = train(X_tr, y_tr, X_v, y_v)\npreds = predict(X_v)\nprint(preds.shape == (20,))","expected_output":"True"},{"description":"predict returns only 0s and 1s","test":"import numpy as np\nnp.random.seed(42)\nX_tr = np.random.randn(100, 10)\ny_tr = np.random.randint(0, 2, 100)\nX_v = np.random.randn(20, 10)\ny_v = np.random.randint(0, 2, 20)\npredict = train(X_tr, y_tr, X_v, y_v)\npreds = predict(X_v)\nprint(set(preds.tolist()).issubset({0, 1}))","expected_output":"True"},{"description":"Deterministic output","test":"import numpy as np\nX_tr = np.random.randn(80, 10)\ny_tr = np.random.randint(0, 2, 80)\nX_v = np.random.randn(20, 10)\ny_v = np.random.randint(0, 2, 20)\nnp.random.seed(99)\np1 = train(X_tr.copy(), y_tr.copy(), X_v.copy(), y_v.copy())\npreds1 = p1(X_v.copy())\nnp.random.seed(99)\np2 = train(X_tr.copy(), y_tr.copy(), X_v.copy(), y_v.copy())\npreds2 = p2(X_v.copy())\nprint(np.array_equal(preds1, preds2))","expected_output":"True"}],"approved_user_libraries":["numpy","sklearn"],"difficulty":"easy","evaluation_metric":"Accuracy","data_info":"Breast Cancer Wisconsin dataset from sklearn: 569 samples with 30 numerical features describing cell nuclei measurements (mean radius, mean texture, mean perimeter, etc.). Binary target: 0 = malignant, 1 = benign. Features are standardized to zero mean, unit variance before being passed to your function.","time_limits":{"test":"60 seconds","dev":"30 seconds"},"schema":"practical_question_v1","min_eval_metric":0.92,"example":{"input":{"X_train":"numpy array of shape (350, 30) -- standardized cell nuclei measurements","y_val":"numpy array of shape (100,) -- validation labels","y_train":"numpy array of shape (350,) -- binary labels (0 = malignant, 1 = benign)","X_val":"numpy array of shape (100, 30) -- standardized validation features"},"output":{"predict":"callable: predict(X) -> numpy array of 0s and 1s, shape (n_samples,)"}},"category":"Logistic Regression","title":"Train a Binary Classifier","model_specs":"User's choice of sklearn model. Harness loads Breast Cancer dataset (30 features, binary target), standardizes features, splits into train/val/test. Evaluated on accuracy on held-out data.","memory_limit":"1 GB","dataset":"Breast Cancer Wisconsin","createdAt":"2026-05-15T12:11:59.896000+00:00","time_limits_seconds":{"dev":30,"test":60},"lower_is_better":false,"gpu":false},{"description":"Implement a **normalization function** for a neural network using only NumPy. Your `normalize(x)` function takes an array of activations and returns normalized outputs.\n\nThe training harness handles everything else (forward pass, backpropagation, weight updates). You just define **how activations are normalized before being passed to the next layer**.\n\nYou may implement **ANY normalization strategy** you like: LayerNorm, RMSNorm, BatchNorm-style, or something creative. Your normalization will be used in a simple neural network trained on MNIST.\n\nNormalization is a critical component in modern LLMs — models like DeepSeek V3, OLMo 2, Gemma 3, and Qwen3 all rely on variants of normalization (especially RMSNorm) to stabilize training and improve performance.","constraints":["Use ONLY numpy - no PyTorch, TensorFlow, or other libraries","normalize(x) must return an array of the SAME shape as input","Must actually change the distribution of values (not just returning x unchanged)","Should work element-wise across the last dimension (feature/hidden dimension)","Must handle batched 2D inputs of shape (batch_size, features)","Output must not contain NaN or Inf values","Any normalization strategy is valid!"],"id":"24","test_cases":[{"description":"Output shape matches input shape","test":"import numpy as np\nx = np.array([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])\ny = normalize(x)\nprint(y.shape == x.shape)","expected_output":"True"},{"description":"Not the identity function","test":"import numpy as np\nx = np.array([[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]])\ny = normalize(x)\nprint(not np.allclose(y, x))","expected_output":"True"},{"description":"Works on different sized inputs","test":"import numpy as np\nx = np.random.randn(8, 128)\ny = normalize(x)\nprint(y.shape == (8, 128))","expected_output":"True"},{"description":"Deterministic output","test":"import numpy as np\nnp.random.seed(42)\nx = np.random.randn(4, 64)\ny1 = normalize(x)\ny2 = normalize(x)\nprint(np.allclose(y1, y2))","expected_output":"True"},{"description":"No NaN or Inf in output","test":"import numpy as np\nx = np.random.randn(16, 32)\ny = normalize(x)\nprint(np.all(np.isfinite(y)))","expected_output":"True"},{"description":"Handles near-zero variance input","test":"import numpy as np\nx = np.ones((4, 16)) * 3.0\nx[0, 0] = 3.001\ny = normalize(x)\nprint(np.all(np.isfinite(y)))","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"easy","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Trained with a 3-layer neural network (784→128→128→10) using Pre-Norm placement. Your normalization is applied 3 times: before each linear layer.","time_limits":{"test":"180 seconds","dev":"90 seconds"},"schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"x":"numpy array of shape (batch_size, features), e.g., [[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0]]"},"output":{"normalized":"numpy array of same shape with normalization applied across the feature dimension, e.g., [[0.365, 0.730, 1.095, 1.461], [0.379, 0.455, 0.530, 0.606]]"},"reasoning":"For RMSNorm: RMS of row [1,2,3,4] = sqrt((1+4+9+16)/4) = sqrt(7.5) ≈ 2.739. Each element is divided by the RMS: [1/2.739, 2/2.739, 3/2.739, 4/2.739] ≈ [0.365, 0.730, 1.095, 1.461]"},"category":"Normalization","title":"Design Your Own Normalization Layer","model_specs":"3-layer neural network (784→128→128→10) with Pre-Norm placement. Your normalization is applied before each linear layer (used 3 times per forward pass), mimicking how RMSNorm is placed in modern LLMs like Llama 3 and DeepSeek V3.","memory_limit":"1 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:42.069000+00:00","function_signature":"normalize(x) -> normalized_x","time_limits_seconds":{"dev":90,"test":180},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a **routing function** for a Mixture of Experts (MoE) layer using only NumPy. Your `route(x, W_gate)` function takes hidden activations and a gate weight matrix, and returns per-expert weights that determine how much each expert contributes to the output.\n\nThe training harness handles everything else: the input layer, expert computations, combining expert outputs using your weights, the output layer, and backpropagation. You just decide **which experts to use and how much**.\n\nThis lab uses **32 experts** with small capacity (16 neurons each) — close to real MoE architectures where many lightweight experts specialize on different inputs. With 32 experts, naive strategies that weight all experts equally perform poorly because gradient signal gets diluted. Smart routing focuses computation on the most relevant experts.\n\nMoE is the most discussed architectural pattern in modern LLMs — DeepSeek V3 uses 256 experts with top-8 routing, Llama 4 uses 128 experts with top-1, and Qwen3 uses 128 experts with top-8.","constraints":["Use ONLY numpy - no PyTorch, TensorFlow, or other libraries","route(x, W_gate) must return an array of shape (batch_size, num_experts)","All weights must be non-negative","Weights for each sample must sum to approximately 1.0","Must not return uniform weights for all inputs (must actually use W_gate)","Must handle batched 2D inputs","Output must not contain NaN or Inf values"],"id":"25","test_cases":[{"description":"Output shape is (batch_size, num_experts)","test":"import numpy as np\nnp.random.seed(42)\nx = np.random.randn(4, 128).astype('float32')\nW_gate = np.random.randn(128, 32).astype('float32') * 0.1\nw = route(x, W_gate)\nprint(w.shape == (4, 32))","expected_output":"True"},{"description":"All weights are non-negative","test":"import numpy as np\nnp.random.seed(42)\nx = np.random.randn(8, 128).astype('float32')\nW_gate = np.random.randn(128, 32).astype('float32') * 0.1\nw = route(x, W_gate)\nprint(np.all(w >= -1e-7))","expected_output":"True"},{"description":"Weights sum to ~1 per sample","test":"import numpy as np\nnp.random.seed(42)\nx = np.random.randn(8, 128).astype('float32')\nW_gate = np.random.randn(128, 32).astype('float32') * 0.1\nw = route(x, W_gate)\nprint(np.allclose(np.sum(w, axis=1), 1.0, atol=1e-4))","expected_output":"True"},{"description":"Not uniform weights (uses W_gate)","test":"import numpy as np\nnp.random.seed(42)\nx = np.random.randn(16, 128).astype('float32')\nW_gate = np.random.randn(128, 32).astype('float32') * 0.1\nw = route(x, W_gate)\nuniform = np.ones_like(w) / 32\nprint(not np.allclose(w, uniform, atol=0.01))","expected_output":"True"},{"description":"Different inputs produce different routing","test":"import numpy as np\nx1 = np.ones((1, 128), dtype='float32') * 0.5\nx2 = -np.ones((1, 128), dtype='float32') * 0.5\nW_gate = np.random.randn(128, 32).astype('float32') * 0.1\nw1 = route(x1, W_gate)\nw2 = route(x2, W_gate)\nprint(not np.allclose(w1, w2))","expected_output":"True"},{"description":"No NaN or Inf in output","test":"import numpy as np\nx = np.random.randn(32, 128).astype('float32') * 10\nW_gate = np.random.randn(128, 32).astype('float32') * 0.5\nw = route(x, W_gate)\nprint(np.all(np.isfinite(w)))","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","time_limits":{"test":"180 seconds","dev":"90 seconds"},"data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Network: Linear(784→128) → ReLU → [MoE layer: 32 experts, each 128→16] → ReLU → Linear(16→10) → Softmax. Your route() function controls how the 32 expert outputs are weighted and combined.","schema":"practical_question_v1","min_eval_metric":0.7,"example":{"input":{"W_gate":"numpy array of shape (hidden_dim, num_experts), e.g., shape (128, 32) — learnable gate weight matrix","x":"numpy array of shape (batch_size, hidden_dim), e.g., shape (64, 128) — hidden activations from the previous layer"},"output":{"weights":"numpy array of shape (batch_size, num_experts), e.g., shape (64, 32) — per-expert weights, non-negative, summing to 1 per row. For top-2 routing, 30 of 32 entries per row would be 0."},"reasoning":"For top-2 sparse gating: compute logits = x @ W_gate (shape 64×32), apply softmax to get scores, find the 2 highest-scoring experts per sample, zero out the remaining 30, and renormalize so weights sum to 1. This activates only 2 of 32 experts per input, making the layer sparse and efficient."},"category":"Mixture of Experts","dataset":"MNIST","title":"Design Your Own MoE Router","model_specs":"3-layer network: Linear(784→128) → ReLU → [MoE: 32 experts, each 128→16] → ReLU → Linear(16→10) → Softmax. Each expert has only 2,064 parameters (128×16 + 16), so routing quality directly determines which small experts get the gradient signal they need to specialize.","memory_limit":"1 GB","createdAt":"2026-05-13T21:01:42.220000+00:00","function_signature":"route(x, W_gate) -> weights","time_limits_seconds":{"dev":90,"test":180},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a `DecisionTree` class. The harness creates 30 instances of your tree, trains each on a different bootstrap sample of the training data, and aggregates their predictions by majority vote to form a Random Forest. You are scored on the **ensemble's** accuracy on a held-out test set, not on any single tree's accuracy.\n\nThe dataset is sklearn's digits dataset (8x8 handwritten digit images, 10 classes, 64 features). Features are standardized before being passed to your tree. To clear the passing threshold, the trees in the ensemble must be **diverse** -- bootstrap sampling alone is not enough. Consider what randomization you can add inside your tree (hint: at each split, only look at a random subset of the features) so that different `random_state` values produce genuinely different trees.\n\nThe harness will also report a single-tree baseline accuracy so you can see the gap your ensemble adds beyond what one tree can do.","type":"general","constraints":["Implement a class named `DecisionTree` with `__init__(self, max_depth=10, min_samples_split=2, max_features='sqrt', random_state=None)`, `fit(self, X, y) -> self`, and `predict(self, X) -> np.ndarray` of integer labels.","`fit` and `predict` must use only numpy. Do NOT use sklearn's tree implementation.","Predictions must be integer class labels (not probabilities).","Two trees constructed with the same `random_state` and trained on the same data must produce identical predictions (determinism).","Trees with different `random_state` values should produce different predictions, given enough features and samples.","Total runtime for fitting 30 trees + a single-tree baseline must fit within the time limit."],"id":"26","test_cases":[{"description":"DecisionTree can be instantiated and fit","test":"import numpy as np\nnp.random.seed(0)\nX = np.random.randn(50, 10); y = np.random.randint(0, 3, 50)\ntree = DecisionTree(max_depth=5, random_state=0)\nresult = tree.fit(X, y)\nprint(result is not None)","expected_output":"True"},{"description":"predict returns a numpy array of correct shape","test":"import numpy as np\nnp.random.seed(0)\nX = np.random.randn(50, 10); y = np.random.randint(0, 3, 50)\ntree = DecisionTree(max_depth=5, random_state=0)\ntree.fit(X, y)\np = tree.predict(np.random.randn(7, 10))\nprint(isinstance(p, np.ndarray) and p.shape == (7,))","expected_output":"True"},{"description":"predict returns valid integer class labels","test":"import numpy as np\nnp.random.seed(0)\nX = np.random.randn(50, 10); y = np.random.randint(0, 3, 50)\ntree = DecisionTree(max_depth=5, random_state=0)\ntree.fit(X, y)\np = tree.predict(np.random.randn(20, 10))\nprint(set(np.unique(p).tolist()).issubset({0, 1, 2}))","expected_output":"True"},{"description":"Trees with different random_state produce different predictions on at least some inputs","test":"import numpy as np\nnp.random.seed(0)\nX = np.random.randn(200, 20); y = np.random.randint(0, 4, 200)\nX_te = np.random.randn(50, 20)\nt1 = DecisionTree(max_depth=8, random_state=1).fit(X, y)\nt2 = DecisionTree(max_depth=8, random_state=999).fit(X, y)\nprint(not np.array_equal(t1.predict(X_te), t2.predict(X_te)))","expected_output":"True"},{"description":"Same random_state and data produce identical predictions (determinism)","test":"import numpy as np\nnp.random.seed(0)\nX = np.random.randn(100, 10); y = np.random.randint(0, 3, 100)\nX_te = np.random.randn(20, 10)\nt1 = DecisionTree(max_depth=6, random_state=42).fit(X, y)\nt2 = DecisionTree(max_depth=6, random_state=42).fit(X, y)\nprint(np.array_equal(t1.predict(X_te), t2.predict(X_te)))","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"hard","evaluation_metric":"ensemble_accuracy","data_info":"sklearn digits dataset: 1797 samples of 8x8 grayscale handwritten digits flattened to 64 features, 10 classes (digits 0-9). Splits: 1200 train / 200 val / 397 test. Features standardized to zero mean, unit variance using training-set statistics.","time_limits":{"test":"180 seconds","dev":"120 seconds"},"schema":"practical_question_v1","lower_is_better":false,"min_eval_metric":0.9,"example":{"input":{"X_train":"numpy array of shape (1200, 64) -- standardized 8x8 digit images flattened to 64 features","y_val":"numpy array of shape (200,) -- validation labels","y_train":"numpy array of shape (1200,) -- integer class labels in [0, 10)","X_val":"numpy array of shape (200, 64) -- standardized validation features"},"output":{"DecisionTree":"class instance with .fit(X, y) -> self and .predict(X) -> np.ndarray of int class labels"}},"category":"Ensembles","title":"Build a Tree for a Random Forest","model_specs":"Harness builds an ensemble of 30 user-defined `DecisionTree` instances, each trained on a fresh bootstrap sample (sampling with replacement, same size as training set) and given a distinct `random_state`. Predictions are aggregated by majority vote. Single-tree baseline (trained on full train set, no bootstrap) is reported for comparison but not scored.","memory_limit":"1 GB","dataset":"sklearn digits","createdAt":"2026-05-13T21:01:42.366000+00:00","time_limits_seconds":{"dev":120,"test":180},"gpu":false},{"description":"The harness has trained five different classifiers (Logistic Regression, Decision Tree, K-Nearest Neighbors, Gaussian Naive Bayes, Random Forest) on a small training set drawn from sklearn's digits dataset (10-class image classification). Your job is to write a function `train_ensemble(base_models, X_val, y_val)` that combines these pre-trained models into an ensemble classifier with higher accuracy than any single one of them.\n\nYou receive the trained base models (each with `.predict` and `.predict_proba`) plus a validation set you can use to fit any meta-parameters of your combiner -- weights, a stacking model, or whatever else you decide. You return a `predict(X)` callable that the harness will run on a held-out test set.\n\nThe win threshold is set deliberately so that naive strategies (returning the best single model, hard voting, soft voting) will not pass. To clear it, your combiner needs to extract information from how the base models *disagree* on the validation set.","type":"general","constraints":["Implement the function `train_ensemble(base_models, X_val, y_val) -> predict_callable`.","`predict` must return a numpy array of integer class labels in [0, 10), with shape (n_samples,).","Do NOT retrain the base models. They are already fit on a separate training set.","You may use any sklearn module to build your combiner (e.g., a meta-learner).","You only have access to `X_val` and `y_val` for tuning. The held-out test set is invisible to you.","Calling the returned `predict` twice on the same input must give identical results."],"id":"27","test_cases":[{"description":"train_ensemble returns a callable","test":"import numpy as np\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nnp.random.seed(0)\nX = np.random.randn(80, 8); y = np.random.randint(0, 3, 80)\nXv = np.random.randn(40, 8); yv = np.random.randint(0, 3, 40)\nmodels = [LogisticRegression(max_iter=500).fit(X, y), DecisionTreeClassifier(max_depth=4, random_state=0).fit(X, y)]\npredict = train_ensemble(models, Xv, yv)\nprint(callable(predict))","expected_output":"True"},{"description":"predict returns numpy array of correct shape","test":"import numpy as np\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nnp.random.seed(0)\nX = np.random.randn(80, 8); y = np.random.randint(0, 3, 80)\nXv = np.random.randn(40, 8); yv = np.random.randint(0, 3, 40)\nmodels = [LogisticRegression(max_iter=500).fit(X, y), DecisionTreeClassifier(max_depth=4, random_state=0).fit(X, y)]\npredict = train_ensemble(models, Xv, yv)\np = predict(np.random.randn(15, 8))\nprint(isinstance(p, np.ndarray) and p.shape == (15,))","expected_output":"True"},{"description":"predict returns valid integer class labels","test":"import numpy as np\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nnp.random.seed(0)\nX = np.random.randn(120, 8); y = np.random.randint(0, 3, 120)\nXv = np.random.randn(60, 8); yv = np.random.randint(0, 3, 60)\nmodels = [LogisticRegression(max_iter=500).fit(X, y), DecisionTreeClassifier(max_depth=4, random_state=0).fit(X, y)]\npredict = train_ensemble(models, Xv, yv)\np = predict(np.random.randn(30, 8))\nprint(set(np.unique(p).tolist()).issubset({0, 1, 2}))","expected_output":"True"},{"description":"predict gives identical results when called twice with same input","test":"import numpy as np\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.tree import DecisionTreeClassifier\nnp.random.seed(0)\nX = np.random.randn(80, 8); y = np.random.randint(0, 3, 80)\nXv = np.random.randn(40, 8); yv = np.random.randint(0, 3, 40)\nmodels = [LogisticRegression(max_iter=500).fit(X, y), DecisionTreeClassifier(max_depth=4, random_state=0).fit(X, y)]\npredict = train_ensemble(models, Xv, yv)\nXt = np.random.randn(20, 8)\np1 = predict(Xt); p2 = predict(Xt)\nprint(np.array_equal(p1, p2))","expected_output":"True"}],"approved_user_libraries":["numpy","sklearn"],"evaluation_metric":"ensemble_accuracy","lower_is_better":false,"data_info":"sklearn digits dataset (1797 samples, 64 features = 8x8 grayscale digit images, 10 classes). Splits: 200 train / 300 val / 1297 test. Features standardized using training-set statistics. Base models are pre-trained on the 200-sample training set.","difficulty":"medium","schema":"practical_question_v1","time_limits":{"test":"120 seconds","dev":"60 seconds"},"min_eval_metric":0.91,"example":{"input":{"y_val":"numpy array of shape (300,) -- integer labels in [0, 10)","X_val":"numpy array of shape (300, 64) -- standardized 8x8 digit features","base_models":"list of 5 trained sklearn-style classifiers (LogisticRegression, DecisionTreeClassifier, KNeighborsClassifier, GaussianNB, RandomForestClassifier), each supporting .predict(X) and .predict_proba(X)"},"output":{"predict":"callable: predict(X) -> numpy array of shape (n,) of integer class labels in [0, 10)"}},"category":"Ensembles","title":"Combine Trained Models into an Ensemble","model_specs":"Five pre-trained sklearn classifiers (LogisticRegression, DecisionTreeClassifier, KNeighborsClassifier, GaussianNB, RandomForestClassifier) are passed to your `train_ensemble` function along with a validation set. Your function returns a `predict` callable; the harness evaluates it on a held-out test set. Reference baselines (best individual, naive hard voting, naive soft voting) are reported alongside your score.","memory_limit":"1 GB","dataset":"sklearn digits","createdAt":"2026-05-13T21:01:42.513000+00:00","time_limits_seconds":{"dev":60,"test":120},"gpu":false},{"description":"You are given a tiny labeled set (20 samples = 2 per class) and a much larger unlabeled set (1300 samples) drawn from sklearn's digits dataset. A classifier trained on the 20 labels alone gets around 65-70% accuracy -- well below the threshold. Your job is to use the unlabeled data to do meaningfully better.\n\nThe standard approach (covered in Chapter 9 of Hands-On ML) is **cluster-based label propagation**: cluster all the data, propagate labels from the labeled samples to entire clusters, and train a downstream classifier on the resulting pseudo-labeled set. The harness reports two reference baselines so you can see what fails: naive logistic regression on the 20 labels, and naive K-means with K=10. Both fall well below the threshold; a correctly designed cluster-then-label pipeline clears it by 5+ points.\n\nYou receive `X_unlabeled`, `X_labeled`, `y_labeled`, `X_val`, `y_val` (validation set for tuning) and return a `predict(X)` callable that the harness applies to a held-out test set.","type":"general","constraints":["Implement `train(X_unlabeled, X_labeled, y_labeled, X_val, y_val) -> predict_callable`.","`predict` must return a numpy array of integer class labels in [0, 10), shape (n_samples,).","Features have already been standardized using statistics from labeled + unlabeled samples. Do NOT re-standardize.","You may use any sklearn module (clustering, classification, manifold learning, mixture models, etc.).","Calling the returned `predict` twice on the same input must give identical results.","You only have access to `X_val` and `y_val` for tuning. The held-out test set is invisible."],"id":"28","test_cases":[{"description":"train returns a callable","test":"import numpy as np\nnp.random.seed(0)\nXu = np.random.randn(100, 8)\nXl = np.random.randn(20, 8); yl = np.random.randint(0, 5, 20)\nXv = np.random.randn(30, 8); yv = np.random.randint(0, 5, 30)\npredict = train(Xu, Xl, yl, Xv, yv)\nprint(callable(predict))","expected_output":"True"},{"description":"predict returns numpy array of correct shape","test":"import numpy as np\nnp.random.seed(0)\nXu = np.random.randn(100, 8)\nXl = np.random.randn(20, 8); yl = np.random.randint(0, 5, 20)\nXv = np.random.randn(30, 8); yv = np.random.randint(0, 5, 30)\npredict = train(Xu, Xl, yl, Xv, yv)\np = predict(np.random.randn(15, 8))\nprint(isinstance(p, np.ndarray) and p.shape == (15,))","expected_output":"True"},{"description":"predict returns valid integer class labels","test":"import numpy as np\nnp.random.seed(0)\nXu = np.random.randn(200, 8)\nXl = np.random.randn(20, 8); yl = np.random.randint(0, 5, 20)\nXv = np.random.randn(30, 8); yv = np.random.randint(0, 5, 30)\npredict = train(Xu, Xl, yl, Xv, yv)\np = predict(np.random.randn(40, 8))\nprint(set(np.unique(p).tolist()).issubset({0, 1, 2, 3, 4}))","expected_output":"True"},{"description":"predict gives identical results when called twice","test":"import numpy as np\nnp.random.seed(0)\nXu = np.random.randn(150, 8)\nXl = np.random.randn(20, 8); yl = np.random.randint(0, 5, 20)\nXv = np.random.randn(30, 8); yv = np.random.randint(0, 5, 30)\npredict = train(Xu, Xl, yl, Xv, yv)\nXt = np.random.randn(20, 8)\np1 = predict(Xt); p2 = predict(Xt)\nprint(np.array_equal(p1, p2))","expected_output":"True"}],"approved_user_libraries":["numpy","sklearn"],"evaluation_metric":"test_accuracy","lower_is_better":false,"data_info":"sklearn digits dataset (1797 samples of 8x8 grayscale digit images, 64 features, 10 classes). Splits: 20 labeled (2 per class, stratified) + 1300 unlabeled + 200 validation + 277 test. Features standardized using statistics computed on labeled + unlabeled data combined.","difficulty":"medium","schema":"practical_question_v1","time_limits":{"test":"120 seconds","dev":"60 seconds"},"min_eval_metric":0.75,"example":{"input":{"y_val":"numpy array of shape (200,) -- validation labels","X_unlabeled":"numpy array of shape (1300, 64) -- standardized 8x8 digit features, no labels","X_labeled":"numpy array of shape (20, 64) -- standardized features (2 per class)","X_val":"numpy array of shape (200, 64) -- standardized validation features","y_labeled":"numpy array of shape (20,) -- integer labels in [0, 10)"},"output":{"predict":"callable: predict(X) -> numpy array of shape (n,) of integer class labels in [0, 10)"}},"category":"Clustering","title":"Few-Shot Classification with Cluster-Based Label Propagation","model_specs":"User's choice of clustering method + downstream classifier. Harness loads digits, creates 2-per-class stratified labeled set + 1300 unlabeled + 200 val + 277 test split, standardizes features, calls user's `train`. Reference baselines reported: naive LR on labels-only and naive K-means with K=10 centroid-to-nearest-labeled propagation.","memory_limit":"1 GB","dataset":"sklearn digits","createdAt":"2026-05-13T21:01:42.654000+00:00","time_limits_seconds":{"dev":60,"test":120},"gpu":false},{"description":"## Fine-Tune DistilGPT2 on TinyStories\n\nYou get a **distilgpt2** model loaded on a T4 GPU, a **tokenizer**, and two lists of short text samples drawn from the [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) dataset (children's-book-simple stories that distilgpt2 has never seen). Your job is to fine-tune the model on **train_texts** so the held-out **val_loss** drops as low as possible — using **any method you like**.\n\n### Your contract\n\nImplement this function:\n\n```python\ndef train(model, tokenizer, train_texts, val_texts):\n    # ... your training logic here ...\n    return model\n```\n\nThat's it. Return the trained model. You **don't need to print anything** — the validator measures val_loss after your training and reports it automatically.\n\n### What you're optimizing\n\nYour score is **val_loss** — mean cross-entropy on a held-out TinyStories slice. **Lower is better.**\n\n- Untrained distilgpt2 baseline: ~4.5 nats\n- Well-fine-tuned: ~3.0 nats\n- **Pass threshold: val_loss ≤ 4.0**\n\n### Recommended approach\n\n**LoRA** is the highest-value method for this lab — small param count, fast on T4, and the Learn tab walks through it in detail. A reference LoRA recipe (r=8 on `c_attn`, AdamW lr=5e-4, 3 epochs, batch=4) reliably hits val_loss ~3.0 in under 90 seconds.\n\nOther viable methods (all admissible):\n\n- **Full fine-tuning** — unfreeze everything, train with lr around 1e-5 to 5e-5.\n- **DoRA** — `LoraConfig(..., use_dora=True)`. Slightly better than LoRA at the same rank.\n- **Prefix / prompt / P-tuning** — via peft.\n- **IA³** — even smaller than LoRA.\n- **Layer freezing** — train only the last block + `lm_head`.\n- **BitFit** — bias-only training (~36k params).\n- **Custom optimizer / schedule** — AdamW + cosine warmup + gradient clipping, etc.\n\n### Compute\n\n- GPU: T4 (16 GB VRAM)\n- Time budget: 60s for Run, 120s for Submit. Model download ~15–20s; dataset stream ~3–5s; training fills the rest.","type":"epoch","constraints":["Implement a top-level function train(model, tokenizer, train_texts, val_texts) and return the trained model.","Do NOT load the dataset yourself — train_texts and val_texts are provided.","Use the model passed to your function (or wrap/replace it inside train() — your call).","You do NOT need to print anything — the validator handles all output formatting.","Time limit: 60s for Run, 120s for Submit. Model + dataset download eat ~20-30s; train accordingly.","No outbound network beyond what transformers and datasets already do (HuggingFace Hub for model + dataset)."],"id":"29","tags":["llm","fine-tuning","gpu","transformers","tinystories","peft","lora"],"test_cases":[{"test":"print(callable(train))","expected_output":"True"},{"test":"import inspect\nprint(len(inspect.signature(train).parameters) == 4)","expected_output":"True"},{"test":"import inspect\nparams = list(inspect.signature(train).parameters)\nprint(params == ['model', 'tokenizer', 'train_texts', 'val_texts'])","expected_output":"True"}],"approved_user_libraries":["torch","transformers","peft","datasets","accelerate","torchao","numpy","pandas","scikit-learn","math","json"],"evaluation_metric":"val_loss","lower_is_better":true,"data_info":"TinyStories is a synthetic dataset of short children's-book-style stories (~250 tokens each), built specifically to train and study tiny language models. The validator streams a slice from the HuggingFace Hub and passes Python lists train_texts and val_texts (list[str]) to your train() function. Validator uses 50 train / 10 val. Grader uses 200 train / 30 val from a different slice.","difficulty":"hard","schema":"train_texts: list[str]  # 50 stories (dev) or 200 stories (test)\nval_texts:   list[str]  # 10 stories (dev) or 30 stories (test)\n# each story is plain UTF-8 text, ~50-400 tokens.","time_limits":{"test":120,"dev":60},"min_eval_metric":4.0,"example":{"input":{"val_texts":["The little dog wagged his tail and barked at the butterfly."],"train_texts":["Once upon a time, there was a little cat named Whiskers. Whiskers loved to play in the garden.","Lily was a happy girl who loved her teddy bear. Every night she hugged it tight."]},"output":{"final_val_acc":3.15}},"category":"LLM Fine-Tuning","title":"Fine-Tune DistilGPT2 on TinyStories","model_specs":"distilgpt2 — 82M params, 6 transformer layers, 768 hidden dim. Loaded in fp16 on cuda. Attention QKV is packed in a single transformers.pytorch_utils.Conv1D module named c_attn (relevant when targeting LoRA).","gpu":true,"memory_limit":"16 GB VRAM (T4)","dataset":"TinyStories (roneneldan/TinyStories) — streamed slice","createdAt":"2026-05-13T21:01:42.779000+00:00","time_limits_seconds":{"dev":60,"test":120}},{"description":"In this open-ended challenge, you will implement your own optimizer in PyTorch. The optimizer can be inspired by existing ones (SGD, RMSProp, Adam, etc.) or be completely new — your design choice.\n\nYou must subclass `torch.optim.Optimizer` and implement the `step()` method. Your optimizer will be tested on a small CNN trained on the MNIST dataset. The goal is to achieve good accuracy (≥80%) within a few epochs.\n\nYou are free to use any update rule, momentum terms, normalization tricks, or bias corrections you like. Creativity is encouraged — as long as the optimizer works and converges on MNIST.","constraints":["Subclass torch.optim.Optimizer.","Implement your own `step()` method that updates parameters.","Dense gradients only (no sparse support required).","The optimizer must be stable enough to train a small CNN on MNIST.","Creativity is encouraged: implement any rule you believe can converge."],"id":"3","test_cases":[{"test":"import torch, torch.nn as nn\nlin = nn.Linear(4,3)\nopt = MyOptimizer(lin.parameters(), lr=1e-2)\nx = torch.randn(8,4); y = torch.randint(0,3,(8,))\nloss = nn.CrossEntropyLoss()(lin(x), y)\nloss.backward();\nwith torch.no_grad():\n    w_before = lin.weight.clone()\nopt.step()\nchanged = (lin.weight - w_before).abs().sum().item()\nprint(changed > 0)\n","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\nfrom copy import deepcopy\n\ntorch.manual_seed(0)\n\n# --- Model & loss ---\nlin = nn.Linear(4, 3, bias=True)\ncriterion = nn.CrossEntropyLoss()\n\n# Save identical starting weights for both cases\ninit_state = deepcopy(lin.state_dict())\n\ndef param_l1_delta(model, before_params):\n    \"\"\"Sum of L1 deltas across ALL params (weights and biases).\"\"\"\n    delta = 0.0\n    for p, b in zip(model.parameters(), before_params):\n        delta += (p.detach() - b).abs().sum().item()\n    return delta\n\ndef snapshot_params(model):\n    return [p.detach().clone() for p in model.parameters()]\n\ndef restore_init(model, state):\n    model.load_state_dict(deepcopy(state))\n\n# --- Case A: Very wrong & large-magnitude inputs (expect LARGE step) ---\nrestore_init(lin, init_state)\nopt_bad = MyOptimizer(lin.parameters(), lr=1e-2)\n\n# Large-norm inputs amplify gradients through the linear layer\nx_bad = torch.randn(32, 4) * 10.0\nwith torch.no_grad():\n    logits_bad = lin(x_bad)\n# Force WRONG targets: shift the argmax by 1 mod 3\ny_bad = (logits_bad.argmax(dim=1) + 1) % 3\n\nopt_bad.zero_grad(set_to_none=True)\nloss_bad = criterion(lin(x_bad), y_bad)\nloss_bad.backward()\n\nbefore_bad = snapshot_params(lin)\nopt_bad.step()\ndiff_bad = param_l1_delta(lin, before_bad)\n\n# --- Case B: Close/correct & small-magnitude inputs (expect SMALLER step) ---\nrestore_init(lin, init_state)\nopt_good = MyOptimizer(lin.parameters(), lr=1e-2)\n\n# Small-norm inputs reduce gradient magnitudes\nx_good = torch.randn(32, 4) * 0.01\nwith torch.no_grad():\n    logits_good = lin(x_good)\n# Choose targets that match current predictions (closer to correct)\ny_good = logits_good.argmax(dim=1)\n\nopt_good.zero_grad(set_to_none=True)\nloss_good = criterion(lin(x_good), y_good)\nloss_good.backward()\n\nbefore_good = snapshot_params(lin)\nopt_good.step()\ndiff_good = param_l1_delta(lin, before_good)\n\nprint(diff_good < diff_bad)\n","expected_output":"True"}],"approved_user_libraries":["torch","numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","time_limits":{"test":"300 seconds","dev":"120 seconds"},"schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"y_sample":7,"X_sample":[{"0":[0.0,0.05,0.1,0.0],"3":[0.0,0.0,0.0,0.0],"1":[0.0,0.12,0.25,0.02],"2":[0.01,0.2,0.6,0.15]}]},"output":{"note":"Example of a small crop from a (1,28,28) MNIST image with label 7. Your optimizer should work on this dataset when training a CNN classifier."}},"category":"Optimizers","title":"MNIST: Design Your Own Pytorch Optimizer","dataset":"MNIST","memory_limit":"2 GB","createdAt":"2026-05-13T21:01:42.922000+00:00","time_limits_seconds":{"dev":120,"test":300},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a custom transformation class `MyTransform` to modify each input tinygrad image tensor used during training. The harness will iterate over MNIST samples one at a time, call your transform on each image, and stack the results into a batch.\n\nTinygrad has no PyTorch-style `DataLoader` class - the framework prefers explicit batching with `Tensor.randint` over `X_train`. This problem teaches the same per-sample transform lesson using a plain Python loop the harness provides.","constraints":["Implement ONLY `MyTransform`.","You MUST return a tinygrad Tensor of the SAME shape and dtype.","The transform must NOT be the identity function.","It must be deterministic under a fixed random seed set via Tensor.manual_seed."],"id":"30","img_type":"tinygrad","test_cases":[{"test":"from tinygrad import Tensor\nx = Tensor.rand(1, 28, 28)\nT = MyTransform()\ny = T(x)\nprint(y.shape == x.shape)","expected_output":"True"}],"approved_user_libraries":["tinygrad","numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","time_limits":{"test":"240 seconds","dev":"90 seconds"},"data_info":"MNIST 28x28 grayscale digits (10 classes). Inputs scaled to [0,1] and shaped (N,1,28,28). Loaded from the Keras MNIST .npz dataset and wrapped as tinygrad Tensors.","schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"y_sample":7,"x_sample":"tinygrad Tensor shape (1, 28, 28), values in [0,1]"},"output":{"note":"Returns a tinygrad Tensor of shape (1, 28, 28) and the same dtype, with statistics measurably different from the input."}},"category":"Data Pipelines","title":"MNIST: Tinygrad Data Transform","model_specs":"Small CNN built as a plain tinygrad class: two Conv2d->ReLU->MaxPool2d stages, then Linear->ReLU->Linear(10). Batch size 128, Adam lr=1e-3, 2 epochs.","memory_limit":"2 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:43.050000+00:00","time_limits_seconds":{"dev":90,"test":240},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a function `build_model()` that returns a tinygrad model (any plain Python class with `__call__`) to classify MNIST digits. You may use ANY tinygrad ops, but if your model has more than **2048 trainable parameters**, your reported accuracy will be **0**. Keep it tiny and clever (e.g., global pooling, grouped/depthwise convs, shared weights, etc.).\n\nTinygrad does **not** use an `nn.Module` base class - your model is just a regular Python class with `__init__` storing submodules as attributes and a `__call__` method that runs the forward pass. Parameters are discovered by walking the class with `tinygrad.nn.state.get_parameters`.","constraints":["Implement ONLY `build_model()`; return any callable Python object that has a `__call__(x) -> Tensor` method.","All trainable Tensor attributes (anything found by `get_parameters`) count toward the 2048 limit.","If param_count > 2048, the grader sets accuracy to 0.","Use only tinygrad APIs (no torch)."],"id":"31","img_type":"tinygrad","test_cases":[{"test":"from tinygrad import Tensor\nm = build_model()\nprint(callable(m))","expected_output":"True"},{"test":"from tinygrad.nn.state import get_parameters\nm = build_model()\nparams = sum(p.numel() for p in get_parameters(m))\nprint(params <= 2048)","expected_output":"True"}],"approved_user_libraries":["tinygrad","numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","time_limits":{"test":"240 seconds","dev":"90 seconds"},"data_info":"MNIST 28x28 grayscale digits (10 classes). Inputs scaled to [0,1] and shaped (N,1,28,28). Loaded from the Keras MNIST .npz dataset and wrapped as tinygrad Tensors.","schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"X":"tinygrad Tensor shape (N, 1, 28, 28), values in [0, 1]","y":"5"},"output":{"note":"If param_count > 2048 -> accuracy is set to 0 in the final JSON.","param_count":"<= 2048"}},"category":"Model Architecture","title":"MNIST: Design-Your-Own Tiny Tinygrad Model","model_specs":"You choose the architecture. Training: Adam lr=0.01, batch size=128, epochs=10. Hard cap: 2048 trainable parameters.","memory_limit":"2 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:43.176000+00:00","time_limits_seconds":{"dev":90,"test":240},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"In this open-ended challenge, you will implement your own optimizer in tinygrad. The optimizer can be inspired by existing ones (SGD, RMSProp, Adam, etc.) or be completely new - your design choice.\n\nYou must subclass `tinygrad.nn.optim.Optimizer` and implement the `schedule_step()` method (the base class's `step()` realizes its returned tensors for you). Your optimizer will be tested on a small CNN trained on MNIST. The goal is to achieve good accuracy (>= 80%) within a few epochs.\n\nYou are free to use any update rule, momentum terms, normalization tricks, or bias corrections you like. Creativity is encouraged - as long as the optimizer works and converges on MNIST.","constraints":["Subclass `tinygrad.nn.optim.Optimizer`.","Implement `schedule_step()` returning the list of Tensors the base class will realize.","Dense gradients only.","The optimizer must be stable enough to train a small CNN on MNIST.","Creativity is encouraged: implement any rule you believe can converge."],"id":"32","img_type":"tinygrad","test_cases":[{"test":"from tinygrad import Tensor, nn\nfrom tinygrad.nn.state import get_parameters\nlin = nn.Linear(4, 3)\nopt = MyOptimizer(get_parameters(lin), lr=1e-2)\nx = Tensor.randn(8, 4)\ny = Tensor.randint(8, high=3)\nTensor.training = True\nw_before = lin.weight.numpy().copy()\nloss = lin(x).sparse_categorical_crossentropy(y)\nloss.backward()\nopt.step()\nimport numpy as np\nchanged = float(np.abs(lin.weight.numpy() - w_before).sum())\nprint(changed > 0)\n","expected_output":"True"},{"test":"from tinygrad import Tensor, nn\nfrom tinygrad.nn.state import get_parameters\nimport numpy as np\n\nTensor.manual_seed(0)\nTensor.training = True\n\n# Same starting weights for both cases.\nlin_init = nn.Linear(4, 3, bias=True)\ninit_w = lin_init.weight.numpy().copy()\ninit_b = lin_init.bias.numpy().copy()\n\ndef param_l1_delta(params, before):\n    return sum(np.abs(p.numpy() - b).sum() for p, b in zip(params, before))\n\ndef snapshot(params):\n    return [p.numpy().copy() for p in params]\n\ndef restore(model, w, b):\n    model.weight = Tensor(w, requires_grad=True)\n    model.bias = Tensor(b, requires_grad=True)\n\n# --- Case A: large-magnitude inputs, wrong labels (expect LARGE step) ---\nlin = nn.Linear(4, 3, bias=True)\nrestore(lin, init_w, init_b)\nopt_bad = MyOptimizer(get_parameters(lin), lr=1e-2)\nx_bad = Tensor.randn(32, 4) * 10.0\nTensor.training = False\nlogits_bad = lin(x_bad)\ny_bad_np = (logits_bad.argmax(axis=1).numpy() + 1) % 3\ny_bad = Tensor(y_bad_np.astype('int64'))\nTensor.training = True\nopt_bad.zero_grad()\nloss_bad = lin(x_bad).sparse_categorical_crossentropy(y_bad)\nloss_bad.backward()\nbefore_bad = snapshot(get_parameters(lin))\nopt_bad.step()\ndiff_bad = param_l1_delta(get_parameters(lin), before_bad)\n\n# --- Case B: small-magnitude inputs, near-correct labels (expect SMALL step) ---\nlin = nn.Linear(4, 3, bias=True)\nrestore(lin, init_w, init_b)\nopt_good = MyOptimizer(get_parameters(lin), lr=1e-2)\nx_good = Tensor.randn(32, 4) * 0.01\nTensor.training = False\nlogits_good = lin(x_good)\ny_good_np = logits_good.argmax(axis=1).numpy()\ny_good = Tensor(y_good_np.astype('int64'))\nTensor.training = True\nopt_good.zero_grad()\nloss_good = lin(x_good).sparse_categorical_crossentropy(y_good)\nloss_good.backward()\nbefore_good = snapshot(get_parameters(lin))\nopt_good.step()\ndiff_good = param_l1_delta(get_parameters(lin), before_good)\n\nprint(diff_good < diff_bad)\n","expected_output":"True"}],"approved_user_libraries":["tinygrad","numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","time_limits":{"test":"300 seconds","dev":"120 seconds"},"schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"y_sample":7,"X_sample":[{"0":[0.0,0.05,0.1,0.0],"3":[0.0,0.0,0.0,0.0],"1":[0.0,0.12,0.25,0.02],"2":[0.01,0.2,0.6,0.15]}]},"output":{"note":"Example of a small crop from a (1,28,28) MNIST image with label 7. Your optimizer should work on this dataset when training a tinygrad CNN classifier."}},"category":"Optimizers","title":"MNIST: Design Your Own Tinygrad Optimizer","dataset":"MNIST","memory_limit":"2 GB","createdAt":"2026-05-13T21:01:43.320000+00:00","time_limits_seconds":{"dev":120,"test":300},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a function `train_step(model, x_batch, y_batch, lr)` that performs ONE step of gradient descent training. You must manually: (1) compute the forward pass, (2) compute the loss, (3) compute gradients using backpropagation, and (4) update the model parameters. Do NOT use `tinygrad.nn.optim` - update weights yourself using the gradients. This teaches how gradient descent actually works in tinygrad.\n\nTinygrad's lazy evaluation means assignments to parameters do not run until you ask for them to be realized. You'll need to call `Tensor.realize(*params)` (or rely on a subsequent `.item()` / `.numpy()` call) to actually materialize the updates.","constraints":["Must compute the forward pass through the model.","Must compute sparse categorical cross-entropy loss.","Must call `loss.backward()` to compute gradients.","Must clear gradients before the backward pass (set `p.grad = None`).","Must update parameters using `p.assign(...)` and realize the updates.","Must return the loss value as a Python float.","Do NOT use `tinygrad.nn.optim` (no SGD, Adam, AdamW, LARS, LAMB, Muon)."],"id":"33","img_type":"tinygrad","test_cases":[{"test":"from tinygrad import Tensor, nn\nmodel = nn.Linear(10, 2)\nx = Tensor.randn(4, 10)\ny = Tensor([0, 1, 0, 1])\nloss = train_step(model, x, y, lr=0.1)\nprint(isinstance(loss, float) and loss > 0)\n","expected_output":"True"},{"test":"from tinygrad import Tensor, nn\nclass TinyMLP:\n    def __init__(self):\n        self.l1 = nn.Linear(10, 5)\n        self.l2 = nn.Linear(5, 2)\n    def __call__(self, x):\n        return self.l2(self.l1(x).relu())\nmodel = TinyMLP()\nx = Tensor.randn(4, 10)\ny = Tensor([0, 1, 0, 1])\nloss = train_step(model, x, y, lr=0.1)\nprint(isinstance(loss, float))\n","expected_output":"True"},{"test":"from tinygrad import Tensor, nn\nfrom tinygrad import nn as _nn\n\n# Detect if tinygrad.nn.optim is used.\n_optim_used = [False]\n_originals = {}\nfor _name in ['SGD', 'Adam', 'AdamW', 'LARS', 'LAMB', 'Muon']:\n    if hasattr(_nn.optim, _name):\n        _originals[_name] = getattr(_nn.optim, _name)\n        def _make_fake(orig, name):\n            def _fake(*a, **kw):\n                _optim_used[0] = True\n                return orig(*a, **kw)\n            return _fake\n        setattr(_nn.optim, _name, _make_fake(_originals[_name], _name))\n\nmodel = nn.Linear(10, 2)\nx = Tensor.randn(4, 10)\ny = Tensor([0, 1, 0, 1])\ntrain_step(model, x, y, lr=0.1)\n\nfor _name, _orig in _originals.items():\n    setattr(_nn.optim, _name, _orig)\nprint(not _optim_used[0])\n","expected_output":"True"}],"approved_user_libraries":["tinygrad","numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Trained with a 3-layer MLP (784 -> 128 -> 128 -> 10) defined as a plain tinygrad class with `__call__`.","time_limits":{"test":"120 seconds","dev":"60 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"x_batch":"tinygrad Tensor of shape (batch_size, features)","model":"plain tinygrad class with __call__","y_batch":"tinygrad Tensor of shape (batch_size,) with class labels (int64)","lr":"float, learning rate (e.g. 0.1)"},"output":{"loss":"float, the loss value for this batch"}},"category":"Tinygrad Fundamentals","title":"Tinygrad: Implement Your Own Gradient Descent Training Step","dataset":"MNIST","memory_limit":"2 GB","createdAt":"2026-05-13T21:01:43.531000+00:00","time_limits_seconds":{"dev":60,"test":120},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement a function `train_model(model, X_train, y_train, X_val, y_val, epochs, batch_size, lr)` that trains a tinygrad model. Use `tinygrad.nn.optim` for optimization and implement proper batching, shuffling, and validation. Return a list of dictionaries containing training metrics for each epoch.","constraints":["Must use `tinygrad.nn.optim` (e.g. `SGD`, `Adam`, `AdamW`) for optimization.","Must implement mini-batch training with the specified batch_size.","Must shuffle training data each epoch.","Must compute validation accuracy and loss after each epoch.","Must return a list of dicts with keys: 'epoch', 'train_loss', 'val_loss', 'val_accuracy'.","Must use `sparse_categorical_crossentropy` for the loss."],"id":"34","img_type":"tinygrad","test_cases":[{"test":"from tinygrad import Tensor, nn\nmodel = nn.Linear(10, 2)\nX_train = Tensor.randn(100, 10)\ny_train = Tensor.randint(100, high=2)\nX_val = Tensor.randn(20, 10)\ny_val = Tensor.randint(20, high=2)\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=2, batch_size=16, lr=0.01)\nprint(isinstance(history, list) and len(history) == 2)\n","expected_output":"True"},{"test":"from tinygrad import Tensor, nn\nmodel = nn.Linear(10, 2)\nX_train = Tensor.randn(100, 10)\ny_train = Tensor.randint(100, high=2)\nX_val = Tensor.randn(20, 10)\ny_val = Tensor.randint(20, high=2)\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=2, batch_size=16, lr=0.01)\nprint(all('epoch' in h and 'train_loss' in h and 'val_loss' in h and 'val_accuracy' in h for h in history))\n","expected_output":"True"},{"test":"from tinygrad import Tensor, nn\nmodel = nn.Linear(10, 2)\nX_train = Tensor.randn(100, 10)\ny_train = Tensor.randint(100, high=2)\nX_val = Tensor.randn(20, 10)\ny_val = Tensor.randint(20, high=2)\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=3, batch_size=16, lr=0.01)\nprint(history[0]['epoch'] == 1 and history[-1]['epoch'] == 3)\n","expected_output":"True"},{"test":"from tinygrad import Tensor, nn\nimport numpy as np\nclass TinyMLP:\n    def __init__(self):\n        self.l1 = nn.Linear(10, 5)\n        self.l2 = nn.Linear(5, 2)\n    def __call__(self, x):\n        return self.l2(self.l1(x).relu())\nmodel = TinyMLP()\nX_train = Tensor.randn(80, 10)\ny_train = Tensor.randint(80, high=2)\nX_val = Tensor.randn(20, 10)\ny_val = Tensor.randint(20, high=2)\nw_before = model.l1.weight.numpy().copy()\nhistory = train_model(model, X_train, y_train, X_val, y_val, epochs=5, batch_size=16, lr=0.1)\nw_after = model.l1.weight.numpy()\nprint(not np.allclose(w_before, w_after))\n","expected_output":"True"}],"approved_user_libraries":["tinygrad","numpy"],"difficulty":"easy","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Trained with a 3-layer MLP (784 -> 128 -> 128 -> 10) defined as a plain tinygrad class with `__call__`.","time_limits":{"test":"120 seconds","dev":"60 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"X_train":"tinygrad Tensor (N, features)","batch_size":"int","y_val":"tinygrad Tensor (M,) int64","epochs":"int","lr":"float","X_val":"tinygrad Tensor (M, features)","model":"plain tinygrad class with __call__","y_train":"tinygrad Tensor (N,) int64"},"output":{"history":"List[Dict] with keys 'epoch', 'train_loss', 'val_loss', 'val_accuracy'"}},"category":"Tinygrad Fundamentals","title":"Tinygrad: Build a Complete Training Loop","dataset":"MNIST","memory_limit":"2 GB","createdAt":"2026-05-13T21:01:43.657000+00:00","time_limits_seconds":{"dev":60,"test":120},"type":"epoch","lower_is_better":false,"gpu":false},{"created_at":"2026-07-29T03:03:59.726000+00:00","description":"# MLP with Dropout and BatchNorm\n\nBuild and train a multi-layer perceptron (MLP) for **binary classification** that uses `nn.BatchNorm1d` and `nn.Dropout` for stable training and regularization.\n\n## Goals\n1. Implement `RegularizedMLP` with **at least two hidden blocks**. Each hidden block must contain, in order: `nn.Linear` -> `nn.BatchNorm1d` -> nonlinearity -> `nn.Dropout`.\n2. The final layer must map to a **single logit** (shape `(N,)` after squeeze) for binary classification.\n3. Implement `train_model` using `nn.BCEWithLogitsLoss` and an Adam optimizer. Train on the tensors the harness provides.\n4. Stay on **CPU only**. Do not call `.cuda()` or `.to(device)`.\n\n## What the harness does\n- Synthesizes a seeded noisy binary-classification dataset (~500 samples, 10 features).\n- Splits into train / held-out test.\n- Instantiates your model, calls `train_model`, then measures test accuracy.\n- Checks that your module graph actually contains `BatchNorm1d` and `Dropout`.\n\n## API you must implement\n```text\nRegularizedMLP(input_dim, hidden_dim=64, dropout_p=0.3) -> nn.Module\nforward(x): x is (N, input_dim) float tensor -> (N,) logits\ntrain_model(model, X_train, y_train, epochs=150, lr=1e-2) -> trained model\n```\n`y_train` is a float tensor of 0/1 labels with shape `(N,)`.","tags":["pytorch","mlp","dropout","batchnorm","regularization","binary-classification"],"constraints":["CPU only: no .cuda(), no .to(device), no GPU assumptions","Only numpy and torch may be imported by learner code","Must use nn.BatchNorm1d and nn.Dropout in the model","Binary logits + BCEWithLogitsLoss (single output unit)","No internet, no sklearn, no file I/O required","Deterministic under harness seeding; keep runtime well under 120s CPU"],"id":"3480fd6b-ee7a-4afd-ba4b-5c934aeab10b","type":"general","lower_is_better":false,"evaluation_metric":"accuracy","approved_user_libraries":["numpy","torch"],"time_limits":{"test":200,"dev":60},"data_info":"Harness synthesizes data on the fly (no files, no download). With seed 42: X ~ N(0,1) of shape (500, 10); labels from a fixed random 2-layer tanh projection of X plus Gaussian noise (std 0.35), thresholded at 0. An 80/20 train/test split is made with a seeded randperm. Validation harness uses 200 samples and fewer epochs.","schema":"Learner receives tensors only via train_model args:\n- X_train: float tensor (N_train, 10)\n- y_train: float tensor (N_train,) with values in {0.0, 1.0}\nModel forward must accept (batch, 10) and return (batch,) logits.\nHarness builds RegularizedMLP(input_dim=10, hidden_dim=64, dropout_p=0.3) in the graded test.","difficulty":"medium","min_eval_metric":0.78,"example":{"input":"model = RegularizedMLP(input_dim=10, hidden_dim=64, dropout_p=0.3)\nlogits = model(torch.randn(16, 10))  # train mode or eval mode","output":"logits.shape == torch.Size([16])  # one logit per sample"},"category":"PyTorch Fundamentals","title":"MLP with Dropout and BatchNorm","gpu":false,"pytorch_starter_code":"import torch\nimport torch.nn as nn\n\n\nclass RegularizedMLP(nn.Module):\n    \"\"\"MLP with BatchNorm1d and Dropout for binary classification.\"\"\"\n\n    def __init__(self, input_dim: int, hidden_dim: int = 64, dropout_p: float = 0.3):\n        super().__init__()\n        # TODO: Build at least two hidden blocks:\n        #   Linear -> BatchNorm1d -> ReLU (or similar) -> Dropout\n        # Final layer: Linear to 1 logit.\n        # Store layers on self (Sequential is fine).\n        pass\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        \"\"\"Return shape (N,) logits for batch x of shape (N, input_dim).\"\"\"\n        # TODO: run x through your network and squeeze the last dim if needed\n        raise NotImplementedError\n\n\ndef train_model(model, X_train, y_train, epochs=150, lr=1e-2):\n    \"\"\"Train model in-place with BCEWithLogitsLoss + Adam. Return model.\"\"\"\n    # TODO:\n    # - criterion = nn.BCEWithLogitsLoss()\n    # - optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n    # - loop epochs: zero_grad -> forward -> loss -> backward -> step\n    # - y_train is float 0/1 with shape (N,)\n    raise NotImplementedError\n","model_specs":"RegularizedMLP: >=2 hidden blocks, each Linear -> BatchNorm1d -> nonlinearity -> Dropout; final Linear to 1 logit; forward returns (N,) logits. train_model: BCEWithLogitsLoss + Adam, full-batch CPU training for `epochs` steps at learning rate `lr`, return the trained model. Module graph must contain nn.BatchNorm1d and nn.Dropout (harness asserts this).","pytorch_solution":"import torch\nimport torch.nn as nn\n\n\nclass RegularizedMLP(nn.Module):\n    \"\"\"MLP with BatchNorm1d and Dropout for binary classification.\"\"\"\n\n    def __init__(self, input_dim: int, hidden_dim: int = 64, dropout_p: float = 0.3):\n        super().__init__()\n        self.net = nn.Sequential(\n            nn.Linear(input_dim, hidden_dim),\n            nn.BatchNorm1d(hidden_dim),\n            nn.ReLU(),\n            nn.Dropout(dropout_p),\n            nn.Linear(hidden_dim, hidden_dim),\n            nn.BatchNorm1d(hidden_dim),\n            nn.ReLU(),\n            nn.Dropout(dropout_p),\n            nn.Linear(hidden_dim, 1),\n        )\n\n    def forward(self, x: torch.Tensor) -> torch.Tensor:\n        return self.net(x).squeeze(-1)\n\n\ndef train_model(model, X_train, y_train, epochs=150, lr=1e-2):\n    criterion = nn.BCEWithLogitsLoss()\n    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n    model.train()\n    for _ in range(epochs):\n        optimizer.zero_grad()\n        logits = model(X_train)\n        loss = criterion(logits, y_train)\n        loss.backward()\n        optimizer.step()\n    return model\n","time_limits_seconds":{"dev":60,"test":200}},{"description":"Design **your own tabular binary classifier** on the real **Adult Census Income** dataset (UCI / OpenML).\n\nPredict whether a person earns **>50K** (class 1, minority). You receive **pandas DataFrames** with **mixed numeric and categorical** columns — raw, with missing values. Build any **classical** pipeline: imputation, encoding, scaling, imbalance handling, model choice, calibration.\n\nReturn `predict_proba(X)` scoring the positive class. Graded on **PR-AUC** (average precision) on a held-out split. Majority-class dummies fail; a careful classical pipeline passes.\n\n**No deep learning.** NumPy / pandas / scikit-learn only.\n","type":"practical","constraints":["Use only numpy, pandas, scipy, and sklearn — no PyTorch, TensorFlow, JAX, or other DL frameworks","Signature: train(X_train, y_train, X_val, y_val) -> predict_proba","X_train / X_val / X for predict_proba are pandas DataFrames with the same columns (mixed numeric + categorical)","predict_proba(X) must return a 1-D numpy array of shape (n_samples,) with finite positive-class scores","y is binary {0, 1} with 1 = income >50K (minority)","Must be deterministic under a fixed numpy random seed","Any classical strategy is valid — creativity encouraged"],"id":"35","test_cases":[{"test":"import numpy as np\nimport pandas as pd\nX_tr = pd.DataFrame({\n    'age': [25, 40, 60, 30, 45, 50, 22, 35, 55, 28, 33, 48],\n    'job': ['a','b','a','b','a','b','a','b','a','b','a','b'],\n    'hrs': [40, 50, 30, 45, 60, 20, 40, 50, 35, 40, 45, 55],\n})\ny_tr = np.array([0,1,1,0,1,0,0,1,1,0,0,1])\nX_v = X_tr.iloc[:4].copy()\ny_v = y_tr[:4]\npp = train(X_tr, y_tr, X_v, y_v)\nprint(callable(pp))\n","description":"Returns a callable on a tiny mixed-type frame","expected_output":"True"},{"test":"import numpy as np\nimport pandas as pd\nrng = np.random.RandomState(0)\nX_tr = pd.DataFrame({\n    'age': rng.randint(18, 80, 40),\n    'job': rng.choice(['a','b','c'], 40),\n    'hrs': rng.randint(10, 80, 40),\n})\ny_tr = (X_tr['age'] > 40).astype(int).to_numpy()\nX_v = X_tr.iloc[:10].copy()\ny_v = y_tr[:10]\npp = train(X_tr, y_tr, X_v, y_v)\ns = np.asarray(pp(X_v), dtype=float).reshape(-1)\nprint(s.shape == (10,) and np.all(np.isfinite(s)))\n","description":"predict_proba shape and finite","expected_output":"True"},{"test":"import numpy as np\nimport pandas as pd\nrng = np.random.RandomState(1)\nn = 60\nX_tr = pd.DataFrame({\n    'age': rng.randint(18, 80, n),\n    'job': rng.choice(['x','y'], n),\n    'hrs': rng.randint(10, 80, n),\n})\ny_tr = ((X_tr['age'] + X_tr['hrs']) > 100).astype(int).to_numpy()\nX_v = X_tr.iloc[:15].copy()\ny_v = y_tr[:15]\npp = train(X_tr, y_tr, X_v, y_v)\ns = np.asarray(pp(X_v), dtype=float).reshape(-1)\nprint(float(np.std(s)) > 1e-8)\n","description":"Scores vary with the input","expected_output":"True"}],"approved_user_libraries":["numpy","pandas","scipy","sklearn"],"evaluation_metric":"PR-AUC","difficulty":"medium","time_limits":{"test":"120 seconds","dev":"90 seconds"},"data_info":"Real UCI Adult Census Income via OpenML (name='adult', version=2). Predict income >50K (minority, ~24%). Mixed numeric + categorical columns with missing values. Dev (Run): stratified subsample of 8000 rows, seed=42. Submit: stratified subsample of 12000 rows, seed=99. Stratified train/val/holdout splits. X is a pandas DataFrame (not pre-encoded).","schema":"practical_question_v1","min_eval_metric":0.55,"example":{"input":{"X_train":"pandas DataFrame — Adult features (age, workclass, education, occupation, ...)","y_val":"numpy array (n_val,)","y_train":"numpy array (n_train,) — 1 if income >50K else 0","X_val":"pandas DataFrame — same columns as X_train"},"output":{"predict_proba":"callable: predict_proba(X: DataFrame) -> numpy array (n,) positive-class scores"}},"category":"Machine Learning","dataset":"Adult Census Income (UCI / OpenML adult v2)","title":"Design Your Own Tabular Classifier","memory_limit":"2 GB","createdAt":"2026-08-05 16:18:09.164988+00:00","model_specs":"Open-ended classical classifier on a real tabular dataset. User implements train(...) -> predict_proba. Harness scores PR-AUC (average precision) on held-out labels. Logistic, SVM, trees, forests, boosting, ensembles all allowed.","time_limits_seconds":{"dev":90,"test":120},"lower_is_better":false,"gpu":false},{"description":"Design **your own post-training quantization (PTQ)** scheme for an MLP trained on real **MNIST**.\n\nMNIST is loaded from the same Google TF/Keras host used by other Deep-ML labs (`storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz`). You get the trained FP32 weights and a calibration batch of real flattened digits in `[0, 1]`.\n\nApply any classical PTQ idea (bit-width, symmetric/asymmetric, per-tensor or per-channel, calibration, GPTQ-style compensation…), then return **dequantized** weights.\n\n**Graded on** mean **cosine similarity** of logits vs the FP32 model on a held-out MNIST split (higher is better).\n\n**PTQ gate:** each tensor must lie on a uniform 4- or 8-bit grid (per-tensor / per-row / per-column) and must not be an FP32 identity copy.\n\n**NumPy only.**\n","type":"practical","constraints":["Use only numpy — no PyTorch, TensorFlow, JAX, or other DL frameworks","Signature: ptq(weights, calib_X) -> dequant_weights dict","Same keys and shapes as input weights; finite floats","Must look like 4/8-bit uniform PTQ (identity FP32 fails)","Must be deterministic under a fixed numpy seed","Any classical PTQ strategy is valid — creativity encouraged"],"id":"36","test_cases":[{"test":"import numpy as np\nrng=np.random.RandomState(0)\nweights={\n 'W1':rng.randn(128,784)*0.02,'b1':rng.randn(128)*0.01,\n 'W2':rng.randn(64,128)*0.05,'b2':rng.randn(64)*0.01,\n 'W3':rng.randn(10,64)*0.05,'b3':rng.randn(10)*0.01,\n}\ndq=ptq(weights, rng.rand(12,784))\nprint(isinstance(dq,dict) and set(dq.keys())==set(weights.keys()) and all(np.asarray(dq[k]).shape==weights[k].shape for k in weights))\n","description":"Correct keys and shapes for MNIST MLP","expected_output":"True"},{"test":"import numpy as np\nrng=np.random.RandomState(1)\nweights={\n 'W1':rng.randn(128,784)*0.05,'b1':rng.randn(128),\n 'W2':rng.randn(64,128)*0.05,'b2':rng.randn(64),\n 'W3':rng.randn(10,64)*0.05,'b3':rng.randn(10),\n}\ndq=ptq(weights, rng.rand(8,784))\nchanged=any(not np.allclose(np.asarray(dq[k]), weights[k]) for k in weights if weights[k].size>=8)\nprint(bool(changed))\n","description":"Not FP32 identity","expected_output":"True"},{"test":"import numpy as np\nrng=np.random.RandomState(2)\nweights={\n 'W1':rng.randn(128,784)*0.03,'b1':rng.randn(128)*0.02,\n 'W2':rng.randn(64,128)*0.05,'b2':rng.randn(64)*0.02,\n 'W3':rng.randn(10,64)*0.05,'b3':rng.randn(10)*0.02,\n}\nX=rng.rand(30,784)\ndq=ptq(weights,X)\n\ndef fwd(w,X):\n    h=np.maximum(X@w['W1'].T+w['b1'],0)\n    h=np.maximum(h@w['W2'].T+w['b2'],0)\n    return h@w['W3'].T+w['b3']\ny1,y2=fwd(weights,X),fwd(dq,X)\nnum=np.sum(y1*y2,axis=1)\nden=np.linalg.norm(y1,axis=1)*np.linalg.norm(y2,axis=1)+1e-12\nprint(bool(np.all(np.isfinite(y2)) and float(np.mean(num/den))>0.95))\n","description":"High cosine on a random batch","expected_output":"True"}],"evaluation_metric":"mean_output_cosine","approved_user_libraries":["numpy"],"data_info":"Real MNIST from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz (same as other Deep-ML labs). Pixels flattened to 784 and scaled to [0,1]. Harness trains MLPClassifier(hidden=(128,64), relu) on a stratified 8k-subset. Dev seed=42; Submit seed=99. User receives FP32 weights + train calibration batch; graded on held-out test logits cosine vs FP32.","time_limits":{"test":"180 seconds","dev":"120 seconds"},"difficulty":"medium","schema":"practical_question_v1","min_eval_metric":0.98,"example":{"input":{"calib_X":"(n_calib, 784) real MNIST train pixels in [0,1]","weights":"dict W1(128,784), b1(128,), W2(64,128), b2(64,), W3(10,64), b3(10,) from MNIST MLP"},"output":{"dequant_weights":"same keys/shapes, low-bit reconstructions as float"}},"category":"Deep Learning","model_specs":"Open-ended weight-only PTQ on an MNIST MLP (784→128→64→10). ptq(weights, calib_X)->dequant_weights. Metric: mean logit cosine vs FP32.","title":"Design Your Own PTQ","dataset":"MNIST","memory_limit":"2 GB","createdAt":"2026-08-05 18:02:12.102041+00:00","time_limits_seconds":{"dev":120,"test":180},"lower_is_better":false,"gpu":false},{"description":"## Train a Tabular Policy with DPO\n\nYou will align a **discrete policy** to pairwise human (synthetic) preferences using **Direct Preference Optimization** — no separate reward model and no PPO loop.\n\n### Setup\n\nThere is a catalog of `n_items` items. Each training example is a pair of indices `(chosen, rejected)` where the chosen item was preferred. Preferences were generated from a hidden Bradley–Terry quality vector with a little label noise.\n\nA frozen **uniform reference** over the catalog is assumed:\n\n$$\\pi_{\\rm ref}(i) = 1/n_{\\rm items}$$\n\nYour job is to learn a policy $\\pi_\\theta$ (or any scoring function) so that preferred items get higher scores on **held-out** pairs.\n\n### What to implement\n\n```python\ndef train(train_chosen, train_rejected, n_items, beta=0.5) -> score\n```\n\n`score(indices)` returns a float array; the grader computes\n\n$$\\mathrm{preference\\_accuracy} = \\mathbb{E}\\big[\\mathbf{1}\\{\\mathrm{score}(y_w) > \\mathrm{score}(y_l)\\}\\big]$$\n\nPass threshold: **0.75**.\n\n### Suggested approach (DPO)\n\nParameterize $\\log\\pi_\\theta$ with logits + log-softmax. Minimize\n\n$$\\mathcal{L}=-\\log\\sigma\\Big(\\beta\\big(\\log\\pi_\\theta(y_w)-\\log\\pi_\\theta(y_l)\\big)\\Big)$$\n\n(the uniform reference cancels in the difference). Gradient descent for a few hundred steps is enough.\n\n### Tips\n\n- Vectorize over the batch of pairs.\n- Clip DPO logits before `exp` for stability.\n- You may use PyTorch if you prefer; numpy is enough.\n","tags":["dpo","rlhf","preference","alignment","numpy"],"constraints":["Implement train(train_chosen, train_rejected, n_items, beta=0.5) and return a callable score(indices).","score(indices) must accept a 1D numpy int array and return a 1D float array of the same shape.","Higher score means more preferred — preference accuracy counts score(chosen) > score(rejected).","No external data downloads. Pure synthetic pairs are provided by the harness.","Time limit: 30s Run, 60s Submit."],"id":"37","type":"epoch","test_cases":[{"test":"import numpy as np\nsc=train(np.array([0,1,2]), np.array([1,2,0]), 3, beta=0.5)\nout=sc(np.array([0,1,2]))\nprint(len(out), int(np.all(np.isfinite(out))))","expected_output":"3 1"}],"lower_is_better":false,"evaluation_metric":"preference_accuracy","data_info":"train() receives integer index pairs (chosen, rejected). A hidden latent quality vector generates labels with light noise. Held-out pairs use the same latent qualities.","approved_user_libraries":["numpy","torch","scipy","scikit-learn","math","json"],"time_limits":{"test":60,"dev":30},"difficulty":"medium","min_eval_metric":0.75,"example":{"input":"n_items=50, 1500 train pairs, beta=0.5","reasoning":"Fit tabular log-probabilities with the DPO logistic loss. The learned scores correlate with the hidden item qualities, so held-out pairs are ranked correctly well above chance (0.5).","output":"preference_accuracy >= 0.75 on held-out pairs"},"category":"Reinforcement Learning","model_specs":"Any model is fine (tabular logits, embeddings, etc.). Score must be comparable across items so that higher score = more preferred.","title":"Train a Tabular Policy with DPO","gpu":false,"memory_limit":"4 GB","dataset":"Synthetic pairwise preferences over a discrete item catalog (Bradley-Terry latent qualities)","time_limits_seconds":{"dev":30,"test":60}},{"created_at":"2026-07-26T16:11:10.364000+00:00","description":"# Implement Train/Val/Test Split and Evaluate a Baseline\n\nIn this lab you will implement the core first step of a supervised ML workflow: a **seeded shuffle split** of a dataset into train, validation, and test sets, plus a **naive mean baseline** evaluated with Mean Absolute Error (MAE) on the held-out test set.\n\n## Your task\n\nImplement:\n\n```python\ndef split_and_baseline(X, y, train_frac, val_frac, test_frac, seed):\n    ...\n```\n\n**Requirements**\n\n1. **Seeded shuffle split**\n   - Use `np.random.default_rng(seed)` and `rng.permutation(n)` on row indices `0 .. n-1`.\n   - Assign contiguous blocks of the permuted indices in order: train, then val, then test.\n   - Sizes: `n_train = int(n * train_frac)`, `n_val = int(n * val_frac)`, and **all remaining rows** go to test (so fractions need not sum in a way that wastes rows).\n   - The three index arrays together must be a partition of `{0, ..., n-1}` (no overlap, full coverage).\n\n2. **Naive baseline**\n   - Fit on the training targets only: `mu = mean(y[train_idx])`.\n   - Predict `mu` for every test example.\n\n3. **Metric**\n   - Return test MAE: `mean(|y_test - mu|)` as a Python float.\n\n4. **Return value**\n   - `(mae, train_idx, val_idx, test_idx)` where the index arrays are 1-D numpy integer arrays.\n\nDo **not** use scikit-learn or any library beyond numpy. Do not fit a real model; the point is the split + baseline pipeline.\n\n## Why this matters\n\nA correct, reproducible split and an honest baseline are the foundation of every supervised project. If the split leaks test rows into training, or the baseline is computed on the wrong fold, later model comparisons are meaningless.","tags":["numpy","train-test-split","baseline","mae","supervised-learning","evaluation"],"constraints":["numpy only (no sklearn, pandas, scipy, torch, or network access)","Must use np.random.default_rng(seed).permutation for the shuffle","n_train = int(n * train_frac), n_val = int(n * val_frac), test gets the remainder","Baseline mean must be computed on y[train_idx] only","Return Python float MAE plus three 1-D integer numpy index arrays","Deterministic given seed; runtime well under 120s"],"id":"3d26c3f9-cb73-4ab1-bc4d-86cfab2af6d1","type":"general","approved_user_libraries":["numpy"],"evaluation_metric":"mae","lower_is_better":true,"time_limits":{"test":120,"dev":60},"difficulty":"easy","schema":"split_and_baseline(X: np.ndarray (n_samples, n_features), y: np.ndarray (n_samples,), train_frac: float, val_frac: float, test_frac: float, seed: int) -> (mae: float, train_idx: np.ndarray[int], val_idx: np.ndarray[int], test_idx: np.ndarray[int]). Index arrays are a partition of range(n_samples).","data_info":"Synthetic linear regression data generated inside the harness with numpy only (no downloads). X ~ N(0,1) with shape (2000, 8) on test / (400, 4) on validation; y = X w + bias + Gaussian noise. Seeded Generator for data and for the learner split.","min_eval_metric":5.0,"example":{"input":"X.shape=(5, 2), y=np.array([1.0, 3.0, 5.0, 7.0, 9.0]), train_frac=0.4, val_frac=0.2, test_frac=0.4, seed=0","output":"mae=<float>, train_idx/val_idx/test_idx partition range(5) with lengths int(5*0.4)=2, int(5*0.2)=1, remainder=2; mae equals mean(|y_test - mean(y_train)|)"},"category":"Machine Learning","model_specs":"No learned model class. Naive baseline: constant predictor equal to the mean of training targets. Evaluate with MAE on the test fold after a seeded shuffle split (train then val then test blocks on a permutation).","gpu":false,"title":"Implement Train/Val/Test Split and Evaluate a Baseline","time_limits_seconds":{"dev":60,"test":120}},{"description":"Implement a NumPy-based `loss_function` that operates on class **probabilities** (softmax outputs) and integer class targets, returning **both** the loss value and the gradient with respect to the probabilities. You may use **any differentiable classification loss** you like.","constraints":["Return both loss and gradient w.r.t. probabilities (same shape as preds)."],"id":"4","test_cases":[{"test":"import numpy as np\npreds = np.array([[0.7, 0.2, 0.1]])\ntarget = np.array([0])\nloss, grad = loss_function(preds, target)\nprint(loss > 0 and grad.shape == preds.shape)","expected_output":"True"},{"test":"import numpy as np\npreds = np.array([[0.9, 0.05, 0.05]])\ntarget = np.array([0])\nloss, grad = loss_function(preds, target)\nwrong_preds = np.array([[0.9, 0.05, 0.05]])\nwrong_target = np.array([1])\nloss, grad = loss_function(preds, target)\nloss_wrong, grad_wrong = loss_function(wrong_preds, wrong_target)\n\nloss_close = loss < loss_wrong\nprint(loss_close)","expected_output":"True"},{"test":"import numpy as np\n# Grad should not be the same for all classes\npreds = np.array([[0.25,0.25,0.25,0.25]])\ntarget = np.array([1])\nloss, grad = loss_function(preds, target)\nprint(np.allclose(grad[0], grad[0,0]))","expected_output":"False"},{"test":"import numpy as np\n# Loss should decrease when prediction confidence for true class increases\npreds_low = np.array([[0.4,0.3,0.3]])\npreds_high = np.array([[0.9,0.05,0.05]])\ntarget = np.array([0])\nloss_low, _ = loss_function(preds_low, target)\nloss_high, _ = loss_function(preds_high, target)\nprint(loss_high < loss_low)\n","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"hard","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits (10 classes). Inputs scaled to [0,1] and reshaped to (N,1,28,28).","time_limits":{"test":"200 seconds","dev":"60 seconds"},"schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"target":[1,0,2,2],"preds":{"0":[0.1,0.7,0.2],"3":[0.05,0.05,0.9],"1":[0.8,0.1,0.1],"2":[0.2,0.3,0.5]}},"output":{"loss_mean":"≈ 0.43","d_preds":{"0":[0.0,-0.3571,0.0],"3":[0.0,0.0,-0.2778],"1":[-0.3125,0.0,0.0],"2":[0.0,0.0,-0.5]}}},"category":"Loss Functions","dataset":"MNIST","title":"MNIST: Classification Loss (with Gradient)","model_specs":"Small CNN: two Conv→ReLU→MaxPool blocks, then Linear→ReLU→Linear(10). Batch size: 128, Optimizer: Adam lr=1e-3, epochs: 2.","memory_limit":"2 GB","createdAt":"2026-05-13T21:01:43.828000+00:00","time_limits_seconds":{"dev":60,"test":200},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement `generate_adversarial_example()` that takes a trained model, an input image, its true label, and a perturbation budget epsilon, then returns a **perturbed image** that (1) fools the model into predicting a different class, (2) stays within epsilon of the original in L∞ norm, and (3) remains in valid pixel range [0,1]. You may use any gradient-based adversarial attack method (FGSM, PGD, etc.).","constraints":["Return tensor of SAME shape as input x","L∞ distance: ||x_adv - x||_∞ ≤ epsilon","Valid range: all pixels in [0,1]","Must fool the model: model(x_adv).argmax() != y","Use gradient-based methods only"],"id":"5","test_cases":[{"test":"import torch\nimport torch.nn as nn\n\n# Dummy model and data\nclass DummyModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.fc = nn.Linear(784, 10)\n    def forward(self, x):\n        return self.fc(x.flatten(1))\n\nmodel = DummyModel()\nmodel.eval()\nx = torch.rand(1, 1, 28, 28)\ny = torch.tensor([3])\nepsilon = 0.1\ncriterion = nn.CrossEntropyLoss()\n\nx_adv = generate_adversarial_example(model, x, y, epsilon, criterion)\nprint(x_adv.shape == x.shape)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\n\nclass DummyModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.fc = nn.Linear(784, 10)\n    def forward(self, x):\n        return self.fc(x.flatten(1))\n\nmodel = DummyModel()\nmodel.eval()\nx = torch.rand(1, 1, 28, 28)\ny = torch.tensor([3])\nepsilon = 0.1\ncriterion = nn.CrossEntropyLoss()\n\nx_adv = generate_adversarial_example(model, x, y, epsilon, criterion)\nlinf_dist = (x_adv - x).abs().max().item()\nprint(linf_dist <= epsilon + 1e-5)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\n\nclass DummyModel(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.fc = nn.Linear(784, 10)\n    def forward(self, x):\n        return self.fc(x.flatten(1))\n\nmodel = DummyModel()\nmodel.eval()\nx = torch.rand(1, 1, 28, 28)\ny = torch.tensor([3])\nepsilon = 0.1\ncriterion = nn.CrossEntropyLoss()\n\nx_adv = generate_adversarial_example(model, x, y, epsilon, criterion)\nin_range = (x_adv.min().item() >= 0.0) and (x_adv.max().item() <= 1.0)\nprint(in_range)","expected_output":"True"}],"approved_user_libraries":["torch","numpy"],"difficulty":"hard","evaluation_metric":"Fooling Rate","data_info":"MNIST 28x28 grayscale digits (10 classes). Test images scaled to [0,1], shaped (1,1,28,28) for single samples.","time_limits":{"test":"240 seconds","dev":"90 seconds"},"schema":"practical_question_v1","min_eval_metric":0.5,"example":{"input":{"y":"tensor([3])","epsilon":"0.1","x":"tensor([[[[0.0, 0.0, ..., 0.8, 0.9], ...]]]) shape (1,1,28,28)"},"output":{"properties":["model(x_adv).argmax() != y","||x_adv - x||_∞ ≤ epsilon","x_adv in [0,1]"],"x_adv":"tensor([[[[0.0, 0.05, ..., 0.85, 1.0], ...]]]) shape (1,1,28,28)"}},"category":"Adversarial Robustness","title":"MNIST: Adversarial Example Generation","model_specs":"Pre-trained CNN provided by the test harness. You do NOT train the model—you attack it.","memory_limit":"2 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:43.953000+00:00","time_limits_seconds":{"dev":90,"test":240},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Build a complete neural network from scratch using **only NumPy**. You must implement forward propagation, backward propagation (backprop), and parameter updates manually. No PyTorch, TensorFlow, or autograd libraries allowed. Implement a simple MLP that achieves ≥85% accuracy on MNIST by manually computing all gradients.","constraints":["Use ONLY numpy - no PyTorch, TensorFlow, JAX, or autograd libraries","Manually implement forward and backward passes","Manually compute gradients for all parameters","Implement at least: Linear layers, ReLU, Softmax, Cross-Entropy loss","Must support mini-batch training"],"id":"6","test_cases":[{"test":"import numpy as np\nnp.random.seed(42)\nmodel = NeuralNetwork(input_size=784, hidden_size=128, output_size=10)\nX = np.random.rand(10, 784)\nprobs = model.forward(X)\nprint(probs.shape == (10, 10) and np.allclose(probs.sum(axis=1), 1.0))","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nmodel = NeuralNetwork(input_size=784, hidden_size=128, output_size=10, lr=0.01)\nX = np.random.rand(32, 784)\ny = np.random.randint(0, 10, 32)\n\n# Get initial predictions\ninitial_probs = model.forward(X)\ninitial_pred = np.argmax(initial_probs, axis=1)\n\n# Train for a few steps\nfor _ in range(10):\n    model.train_step(X, y)\n\n# Get updated predictions\nfinal_probs = model.forward(X)\nfinal_pred = np.argmax(final_probs, axis=1)\n\n# Check that predictions changed (learning happened)\nchanged = not np.array_equal(initial_pred, final_pred)\nprint(changed)","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nmodel = NeuralNetwork(input_size=784, hidden_size=128, output_size=10, lr=0.1)\nX = np.random.rand(32, 784)\ny = np.random.randint(0, 10, 32)\n\n# Training should decrease loss\ninitial_loss = model.train_step(X, y)\nfor _ in range(20):\n    loss = model.train_step(X, y)\nfinal_loss = loss\n\nprint(final_loss < initial_loss)","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"hard","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits (10 classes). Inputs flattened to 784-dim vectors, scaled to [0,1].","time_limits":{"test":"300 seconds","dev":"120 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"X_batch":"np.array shape (32, 784), values in [0,1]","y_batch":"np.array shape (32,), integer labels 0-9"},"output":{"gradients":"Computed via manual backprop through all layers","forward_output":"np.array shape (32, 10), class probabilities"}},"category":"Neural Network Fundamentals","title":"MNIST: Build Neural Network from Scratch (NumPy Only)","model_specs":"MLP architecture: Input(784) → Hidden(128, ReLU) → Output(10, Softmax). You implement everything from scratch.","memory_limit":"2 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:44.123000+00:00","time_limits_seconds":{"dev":120,"test":300},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"You're given a **very deep network** (30 layers) that suffers from training problems, achieving only ~20% accuracy. Your task: **fix the architecture** to achieve high accuracy while **maintaining the depth (≥30 layers)**. The baseline deep network trains poorly due to vanishing gradients. You can modify the architecture however you like (skip connections, normalization, etc.) but you must keep at least 30 layers. Solutions with <30 layers automatically score 0.","constraints":["Network must have ≥30 layers","Solutions with <30 layers automatically score 0","Must train on CPU within 2 minutes","Any architecture modifications allowed","Can add skip connections, normalization, change activations, etc.","Final score = validation accuracy if depth ≥30, else 0"],"scoring_formula":"If layers < 30: score=0. Else: score = val_accuracy. Rank by highest accuracy.","id":"7","test_cases":[{"test":"import torch\nimport torch.nn as nn\n\ntorch.manual_seed(42)\nmodel = DeepNetwork(input_size=784, hidden_size=128, num_classes=10)\nx = torch.randn(16, 784)\n\ntry:\n    output = model(x)\n    print(output.shape == torch.Size([16, 10]))\nexcept:\n    print(False)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\n\ntorch.manual_seed(42)\nmodel = DeepNetwork(input_size=784, hidden_size=128, num_classes=10)\n\n# Count Linear layers in the main stack\ndef count_layers(model):\n    count = 0\n    # Check model.layers (plain architecture)\n    if hasattr(model, 'layers') and isinstance(model.layers, nn.ModuleList):\n        for module in model.layers:\n            if isinstance(module, nn.Linear):\n                count += 1\n    # Check model.blocks (residual architecture)\n    if hasattr(model, 'blocks') and isinstance(model.blocks, nn.ModuleList):\n        for block in model.blocks:\n            for module in block.modules():\n                if isinstance(module, nn.Linear):\n                    count += 1\n    return count\n\nlayer_count = count_layers(model)\nprint(layer_count >= 30)","expected_output":"True"},{"test":"import torch\nimport torch.nn as nn\n\ntorch.manual_seed(42)\nmodel = DeepNetwork(input_size=784, hidden_size=128, num_classes=10)\nx = torch.randn(16, 784, requires_grad=True)\n\ntry:\n    output = model(x)\n    loss = output.sum()\n    loss.backward()\n    # Check that gradients flow through the network\n    has_grad = x.grad is not None and x.grad.abs().sum() > 0\n    print(has_grad)\nexcept:\n    print(False)","expected_output":"True"}],"approved_user_libraries":["torch","numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits (10 classes). Inputs flattened to 784-dim vectors, scaled to [0,1].","time_limits":{"test":"180 seconds","dev":"120 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"X":"np.array shape (batch, 784), values in [0,1]","y":"np.array shape (batch,), integer labels 0-9"},"output":{"val_accuracy":"Baseline gets ~20%, you should get 90%+","note":"Network must remain deep (≥30 layers)"}},"category":"Network Optimization","title":"MNIST: Fix Very Deep Network Training","model_specs":"Baseline: 30-layer plain network (trains poorly). You must keep ≥30 layers in your fix.","memory_limit":"2 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:44.258000+00:00","time_limits_seconds":{"dev":120,"test":180},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement an **optimizer update rule** using only NumPy. You will write a function `optimizer_step(param, grad, state, lr)` that takes a parameter array and its gradient, then returns the updated parameter.\n\nThe training harness handles forward/backward passes and gives you the gradients—you just decide **how to update the weights**.\n\nYou may implement **ANY update rule** you like: vanilla SGD, momentum, RMSprop, Adam, or something creative. Your optimizer will train a simple neural network on MNIST.","constraints":["Use ONLY numpy - no PyTorch, TensorFlow, or autograd libraries","optimizer_step() receives the gradient already computed","Return updated parameter with the SAME shape as input","Use the state dict to store momentum, running averages, etc.","Any update rule is valid - creativity encouraged!"],"id":"8","test_cases":[{"test":"import numpy as np\nnp.random.seed(42)\nparam = np.array([1.0, 2.0, 3.0])\ngrad = np.array([0.1, 0.2, 0.3])\nstate = {}\nnew_param, new_state = optimizer_step(param, grad, state, lr=0.1)\nprint(new_param.shape == param.shape)","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nparam = np.array([1.0, 2.0, 3.0])\ngrad = np.array([0.1, 0.2, 0.3])\nstate = {}\nnew_param, new_state = optimizer_step(param, grad, state, lr=0.1)\n# Parameter should change when gradient is non-zero\nchanged = np.sum(np.abs(new_param - param)) > 0.001\nprint(changed)","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nparam = np.array([1.0, 2.0, 3.0])\ngrad = np.array([0.0, 0.0, 0.0])  # Zero gradient\nstate = {}\nnew_param, new_state = optimizer_step(param, grad, state, lr=0.1)\n# With zero gradient (and no momentum), param should stay similar\nprint(np.allclose(new_param, param, atol=0.01))","expected_output":"True"},{"test":"import numpy as np\nnp.random.seed(42)\nparam = np.array([[1.0, 2.0], [3.0, 4.0]])\ngrad = np.array([[0.1, 0.2], [0.3, 0.4]])\nstate = {}\nnew_param, new_state = optimizer_step(param, grad, state, lr=0.1)\n# Should work with 2D arrays too\nprint(new_param.shape == (2, 2))","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"medium","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Trained with a 2-layer neural network (784→128→10).","time_limits":{"test":"180 seconds","dev":"90 seconds"},"schema":"practical_question_v1","min_eval_metric":0.85,"example":{"input":{"param":"numpy array of any shape (e.g., [784, 128] weight matrix)","grad":"numpy array of same shape (gradient of loss w.r.t. param)","state":"dict to store momentum, running averages, step count, etc.","lr":"learning rate (e.g., 0.01)"},"output":{"new_param":"updated parameter array (same shape)","state":"updated state dict"}},"category":"Optimizers","title":"Design Your Own Optimizer (NumPy)","model_specs":"2-layer neural network (784→128→10) with ReLU activation. You implement only the parameter update rule.","memory_limit":"1 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:44.402000+00:00","time_limits_seconds":{"dev":90,"test":180},"type":"epoch","lower_is_better":false,"gpu":false},{"description":"Implement an **activation function** for a neural network using only NumPy. Your `activation(x)` function takes an array of values and returns the activated outputs.\n\nThe training harness handles everything else (forward pass, backpropagation, weight updates). You just define **how neurons transform their inputs**.\n\nYou may implement **ANY activation function** you like: ReLU, sigmoid, tanh, or something creative. Your activation will be used in a simple neural network trained on MNIST.","constraints":["Use ONLY numpy - no PyTorch, TensorFlow, or other libraries","activation(x) must return an array of the SAME shape as input","Must be a non-linear function (not just returning x unchanged)","Should work element-wise on arrays of any shape","Any activation function is valid!"],"id":"9","test_cases":[{"test":"import numpy as np\nx = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])\ny = activation(x)\nprint(y.shape == x.shape)","expected_output":"True"},{"test":"import numpy as np\nx = np.array([-2.0, -1.0, 0.0, 1.0, 2.0])\ny = activation(x)\n# Should not be identity function\nprint(not np.allclose(y, x))","expected_output":"True"},{"test":"import numpy as np\nx = np.array([[1.0, -1.0], [2.0, -2.0]])\ny = activation(x)\n# Should work on 2D arrays\nprint(y.shape == (2, 2))","expected_output":"True"},{"test":"import numpy as np\n# Should be deterministic\nx = np.array([1.0, 2.0, 3.0])\ny1 = activation(x)\ny2 = activation(x)\nprint(np.allclose(y1, y2))","expected_output":"True"}],"approved_user_libraries":["numpy"],"difficulty":"easy","evaluation_metric":"Accuracy","data_info":"MNIST 28x28 grayscale digits flattened to 784-dim vectors, scaled to [0,1]. Trained with a 3-layer neural network (784→128→128→10).","time_limits":{"test":"180 seconds","dev":"90 seconds"},"schema":"practical_question_v1","min_eval_metric":0.8,"example":{"input":{"x":"numpy array of any shape, e.g., [-2.0, -1.0, 0.0, 1.0, 2.0]"},"output":{"activated":"numpy array of same shape with activation applied element-wise"}},"category":"Activation Functions","title":"Design Your Own Activation Function","model_specs":"3-layer neural network (784→128→128→10). Your activation is applied after each hidden layer (used twice).","memory_limit":"1 GB","dataset":"MNIST","createdAt":"2026-05-13T21:01:44.529000+00:00","time_limits_seconds":{"dev":90,"test":180},"type":"epoch","lower_is_better":false,"gpu":false},{"created_at":"2026-07-29T03:03:59.885000+00:00","description":"# Train a Tiny CNN Image Classifier\n\nBuild and train a small convolutional neural network to classify tiny synthetic images into two pattern classes (horizontal vs vertical stripes).\n\n## What you implement\n\n1. **`build_model(img_size=8, n_classes=2)`**  \n   Return a `torch.nn.Module` CNN with at least: `Conv2d -> ReLU -> pool -> Linear` (you may add a second conv block). Input images are single-channel `(N, 1, img_size, img_size)`.\n\n2. **`train_model(model, train_x, train_y, epochs=15, lr=0.01, batch_size=32, seed=0)`**  \n   Train `model` in place on the provided tensors with a standard classification loss and an optimizer. Return the trained model. Keep everything on CPU (no `.cuda()` / `.to(device)`).\n\n## Data (provided by the harness)\n\n- `train_x`: `FloatTensor` shape `(N_train, 1, H, W)`\n- `train_y`: `LongTensor` shape `(N_train,)` with labels in `{0, 1}`\n- Test split is held by the harness; it will call your APIs, then measure accuracy.\n\n## Success criteria\n\nTest accuracy must clearly beat chance on this easy synthetic task (see grader floor). Seed all randomness you introduce so runs are deterministic.","tags":["pytorch","cnn","convolution","image-classification","training","pooling"],"constraints":["CPU only: do not call .cuda(), .to('cuda'), or .to(device)","Use only numpy and torch (no sklearn, no torchvision datasets, no network I/O)","Implement build_model and train_model with the specified signatures","Model must include Conv2d -> ReLU -> pool -> Linear (extras allowed)","Seed randomness inside train_model when shuffling","Return logits of shape (N, n_classes); harness uses argmax for accuracy","Keep total runtime well under the time limit on CPU"],"id":"91c76ab2-7d3a-43a7-98ce-385aab6f1691","type":"general","lower_is_better":false,"evaluation_metric":"accuracy","approved_user_libraries":["numpy","torch"],"time_limits":{"test":200,"dev":60},"data_info":"Harness synthesizes a seeded two-class dataset with numpy/torch only (no downloads). Class 0: horizontal stripe patterns on an 8x8 grid (random 0/1 phase). Class 1: vertical stripes. Independent Gaussian noise (std~0.12) is added per image. Default split: 320 train / 80 test, balanced classes, shuffled with RandomState(42). Pixel values are float32; labels int64.","schema":"Learner APIs receive: train_x FloatTensor (N, 1, H, W) with H=W=img_size (default 8); train_y LongTensor (N,) in {0,1}. build_model(img_size, n_classes) -> nn.Module. train_model(model, train_x, train_y, epochs, lr, batch_size, seed) -> trained nn.Module. Harness owns test_x/test_y and computes accuracy.","difficulty":"hard","min_eval_metric":0.85,"example":{"input":"build_model(img_size=8, n_classes=2) then train_model(model, train_x, train_y, epochs=12) where train_x is FloatTensor (320, 1, 8, 8) and train_y is LongTensor (320,) with labels 0/1 (horizontal vs vertical stripes + light noise).","output":"Trained nn.Module; harness evaluates on held-out test set and reports JSON like {\"test_acc\": 0.975}"},"category":"Model Architecture","title":"Train a Tiny CNN Image Classifier","gpu":false,"pytorch_starter_code":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\n\n\nclass TinyCNN(nn.Module):\n    \"\"\"Small CNN: Conv2d -> ReLU -> pool -> (optional extras) -> Linear.\"\"\"\n\n    def __init__(self, img_size=8, n_classes=2):\n        super().__init__()\n        # TODO: define conv, activation, pooling, and classifier layers\n        # Input shape: (N, 1, img_size, img_size)\n        raise NotImplementedError\n\n    def forward(self, x):\n        # TODO: implement forward pass; return logits of shape (N, n_classes)\n        raise NotImplementedError\n\n\ndef build_model(img_size=8, n_classes=2):\n    \"\"\"Return an instance of your TinyCNN (or equivalent nn.Module).\"\"\"\n    # TODO: return TinyCNN(img_size=img_size, n_classes=n_classes)\n    raise NotImplementedError\n\n\ndef train_model(model, train_x, train_y, epochs=15, lr=0.01, batch_size=32, seed=0):\n    \"\"\"Train model on train_x/train_y and return the trained model.\n\n    Args:\n        model: nn.Module from build_model\n        train_x: FloatTensor (N, 1, H, W)\n        train_y: LongTensor (N,)\n        epochs: number of full passes over the data\n        lr: optimizer learning rate\n        batch_size: mini-batch size\n        seed: RNG seed for shuffling / init determinism\n\n    Returns:\n        Trained model (same instance is fine).\n    \"\"\"\n    # TODO:\n    # - torch.manual_seed(seed)\n    # - CrossEntropyLoss + Adam (or SGD)\n    # - mini-batch loop for `epochs` epochs\n    # - model.train(); zero_grad -> forward -> loss -> backward -> step\n    raise NotImplementedError\n","model_specs":"CPU-only torch.nn.Module CNN. Required pattern: at least one Conv2d, ReLU, spatial pool (e.g. MaxPool2d), then Linear classifier to n_classes logits. Single-channel input. No Dropout/BatchNorm required. No .cuda() or .to(device). Cross-entropy training with a torch optimizer for several epochs.","pytorch_solution":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\n\n\nclass TinyCNN(nn.Module):\n    \"\"\"Small CNN: Conv2d -> ReLU -> pool -> Linear.\"\"\"\n\n    def __init__(self, img_size=8, n_classes=2):\n        super().__init__()\n        self.features = nn.Sequential(\n            nn.Conv2d(1, 8, kernel_size=3, padding=1),\n            nn.ReLU(),\n            nn.MaxPool2d(2),\n        )\n        flat_dim = 8 * (img_size // 2) * (img_size // 2)\n        self.classifier = nn.Linear(flat_dim, n_classes)\n\n    def forward(self, x):\n        x = self.features(x)\n        x = x.view(x.size(0), -1)\n        x = self.classifier(x)\n        return x\n\n\ndef build_model(img_size=8, n_classes=2):\n    return TinyCNN(img_size=img_size, n_classes=n_classes)\n\n\ndef train_model(model, train_x, train_y, epochs=15, lr=0.01, batch_size=32, seed=0):\n    torch.manual_seed(seed)\n    criterion = nn.CrossEntropyLoss()\n    optimizer = optim.Adam(model.parameters(), lr=lr)\n    model.train()\n    n = train_x.size(0)\n    for _ in range(epochs):\n        perm = torch.randperm(n)\n        for i in range(0, n, batch_size):\n            idx = perm[i:i + batch_size]\n            bx = train_x[idx]\n            by = train_y[idx]\n            optimizer.zero_grad()\n            logits = model(bx)\n            loss = criterion(logits, by)\n            loss.backward()\n            optimizer.step()\n    return model\n","time_limits_seconds":{"dev":60,"test":200}},{"created_at":"2026-07-29T03:03:59.579000+00:00","description":"# Fit Linear Regression with Autograd\n\nImplement ordinary least squares linear regression using **only** PyTorch autograd. Do **not** use `torch.optim` or closed-form solvers.\n\n## Task\n\nWrite `fit_linear_regression(X, y, lr=0.1, steps=500)` that:\n\n1. Initializes parameters `w` (shape `(D,)`) and `b` (scalar) with `requires_grad=True`.\n2. Runs full-batch gradient descent for `steps` iterations minimizing mean squared error (MSE):\n   `L = mean((X @ w + b - y)^2)`.\n3. Uses autograd (`loss.backward()`) to obtain gradients and manually updates parameters with the learning rate `lr`.\n4. Returns the learned `(w, b)` as tensors **without** gradient tracking (detached).\n\nThe harness builds a seeded synthetic dataset `y = X @ w_true + b_true + noise`, holds out a test split, calls your function on the train split, and scores **test MSE** (lower is better).\n\n## Rules\n\n- Use CPU tensors only (no `.cuda()` / `.to(device)`).\n- Only `numpy` and `torch` are available.\n- Do not use `torch.optim`, `torch.nn`, or analytic normal equations.","tags":["pytorch","autograd","gradients","linear-regression","mse"],"constraints":["CPU-only tensors; no .cuda() or .to(device)","Do not use torch.optim or torch.nn","Do not use closed-form / normal-equation solutions","Only numpy and torch may be imported","Must use autograd (loss.backward) for gradients","Return detached tensors for w and b","Keep runtime well under 120s on CPU"],"id":"9ff596ea-672e-4101-9ce4-0856c55b62c9","type":"general","lower_is_better":true,"evaluation_metric":"mse","approved_user_libraries":["numpy","torch"],"time_limits":{"test":200,"dev":60},"data_info":"Harness synthesizes data with torch.manual_seed(42): X ~ N(0,1) of shape (200, 3), w_true=[1.5,-2.0,0.5], b_true=0.3, y = X @ w_true + b_true + 0.1*N(0,1). A seeded randperm holds out 40 samples for test; the learner fits on the remaining 160.","schema":"fit_linear_regression(X, y, lr=0.1, steps=500) receives X: float tensor (N, D), y: float tensor (N,) or (N, 1). Returns w: float tensor (D,), b: scalar float tensor. Test harness uses D=3.","difficulty":"medium","min_eval_metric":0.25,"example":{"input":"X shape (160, 3), y shape (160,), lr=0.1, steps=500 on seeded y = X @ [1.5, -2.0, 0.5] + 0.3 + noise","output":"w ~= [1.5, -2.0, 0.5], b ~= 0.3; held-out MSE typically near noise floor (~0.01)"},"category":"PyTorch Fundamentals","title":"Fit Linear Regression with Autograd","gpu":false,"pytorch_starter_code":"import torch\nimport numpy as np\n\n\ndef fit_linear_regression(X, y, lr=0.1, steps=500):\n    \"\"\"Fit y ~= X @ w + b with full-batch GD using only autograd.\n\n    Args:\n        X: Float tensor (N, D)\n        y: Float tensor (N,) or (N, 1)\n        lr: learning rate\n        steps: number of gradient descent iterations\n\n    Returns:\n        w: Float tensor (D,) learned weights (no grad)\n        b: Float tensor scalar learned bias (no grad)\n    \"\"\"\n    # TODO: ensure y is shape (N,)\n    # TODO: initialize w (D,) and b with requires_grad=True\n    # TODO: for each step:\n    #   - predict, compute MSE loss\n    #   - loss.backward()\n    #   - manual GD update under torch.no_grad()\n    #   - zero gradients\n    # TODO: return detached w, b\n    raise NotImplementedError\n","model_specs":"Manual full-batch gradient descent linear regressor: parameters w (D,) and b (scalar) with requires_grad=True; MSE loss; autograd backward; in-place GD updates only (no torch.optim, no torch.nn modules).","pytorch_solution":"import torch\nimport numpy as np\n\n\ndef fit_linear_regression(X, y, lr=0.1, steps=500):\n    \"\"\"Fit y ~= X @ w + b with full-batch GD using only autograd.\n\n    Args:\n        X: Float tensor (N, D)\n        y: Float tensor (N,) or (N, 1)\n        lr: learning rate\n        steps: number of gradient descent iterations\n\n    Returns:\n        w: Float tensor (D,) learned weights (no grad)\n        b: Float tensor scalar learned bias (no grad)\n    \"\"\"\n    y = y.reshape(-1)\n    N, D = X.shape\n\n    w = torch.zeros(D, dtype=X.dtype, requires_grad=True)\n    b = torch.zeros((), dtype=X.dtype, requires_grad=True)\n\n    for _ in range(steps):\n        pred = X @ w + b\n        loss = ((pred - y) ** 2).mean()\n        loss.backward()\n\n        with torch.no_grad():\n            w -= lr * w.grad\n            b -= lr * b.grad\n            w.grad.zero_()\n            b.grad.zero_()\n\n    return w.detach(), b.detach()\n","time_limits_seconds":{"dev":60,"test":200}}]}