Skip to content
Insights · AI Agents

Which category did that call flag?

Name the category that call flagged, and the score beside it. A different call can flag another category, and the model does not get to recall it.

Scroll
AI AgentsOct 30, 202613 min readBy Salman Naqvi, Founder & CEO
Which category did that call flag?

Do not treat the reply as something a filter cleared, and do not ask the model whether it was safe, until you can name the moderation call. Tonight, three proofs, in the log you stored. Name the category that call flagged: the moderation id, the model, the category, the boolean, and the score beside it, or a written line that the call was not stored. Show a different call whose flagged category, or whose score, is not this one. Show that you stored the response rather than asking the model to recall it. If any of the three needs the person who wired the request and then left, you have not named the category. You have a sentence and a guess.

This is the moderation object on one call. It is not why the tokens stopped. Name the stop the response carried is stop_reason. A completed turn can still have flagged a category, and a flagged category does not say the tokens ended. It is not the tool result. Which tool result did the model see? is the result stored for a tool call. A tool result is not a category boolean. It is not the thinking budget. Which budget did that thinking use? is the budget you sent. A budget is not a score between zero and one. It is not the page a citation named. Open the page the answer cited is a source the answer pointed at. A URL is not a moderation category. It is not the prompt. Which prompt did that request send? is the text you sent. The text can be what was classified, and the text is still not the boolean.

None of what follows is a rate for this product, and none of it is legal advice. The objects below are one vendor's current moderation examples, and the two examples do not share a score. The figures below that are a 2022 paper about a classifier trained on a taxonomy that is not this API's list. An AUPRC on a production validation set is not the boolean on the call you stored tonight. A paper that never saw this request cannot name the category.

The first example on the moderation page is a placeholder, and it still has a shape you can misread. id is the string id. model is the string model. flagged is true. Every category boolean is true, and every category_scores value is 0. 0 is not true. A score of 0 beside a boolean of true is what that example prints. It is not a threshold, and it is not a cleared call. flagged true is not a category name. The request beside that example sends model omni-moderation-2024-09-26. The response does not print that string. It prints model. Do not paste the request's model onto the response. The applied input types in that example are text on every category. text is not image. There are 13 category keys: harassment, harassment/threatening, hate, hate/threatening, illicit, illicit/violent, self-harm, self-harm/instructions, self-harm/intent, sexual, sexual/minors, violence, and violence/graphic. 13 keys. The shorthand on the same page says harassment, harassment/threatening, hate, and 10 more. 3 named plus 10 more is 13. The 10 more is not a fourteenth category (OpenAI, Moderations).

The second example is a different object, and it does not rescue the first. id is modr-0d9740456c391e43c445bf0f010940c7. 37 characters. The string id is 2 characters. 37 is not 2. model is omni-moderation-latest, which matches the request beside it and does not match the first response's model. flagged is true again. harassment is true with score 0.8189693396524255. harassment/threatening is true with score 0.804985420696006. 0.8189693396524255 is not 0.804985420696006, and neither is 0 from the first example. violence is true with score 0.9999992735124786. violence/graphic is true with score 0.843064871157054. 0.9999992735124786 is not 1. sexual is false with score 1.573112165348997e-6. sexual/minors is false with score 2.212566909570261e-8. A number written with an e is not the integer 0, and 1.573112165348997e-6 is not 2.212566909570261e-8. hate is false with score 0.007562942636942845. In the first example hate is true. false is not true. The same split holds for sexual, illicit, and self-harm. Do not copy a boolean from the example that was convenient.

Applied input types are a third field, and the two examples do not share it. In the second example, harassment is text only. sexual is text and image. self-harm is text and image. violence is text and image. sexual/minors is text only. illicit is text only. text is not image, and a category that lists both is not a category that lists one. The model parameter documents four names: omni-moderation-latest, omni-moderation-2024-09-26, text-moderation-latest, and text-moderation-stable. 4 names. The default in the parameter text is omni-moderation-latest. The first request sends omni-moderation-2024-09-26. 4 is not 13, and a model name is not a category. illicit and illicit/violent can be boolean or null. null is not false. The page's definition of hate says hateful content aimed at non-protected groups, for example chess players, is harassment. That sentence is a definition. It is not a score, and chess players is not a category key.

The paper that built an earlier moderation system used eight labels, not these thirteen keys. Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng describe five top-level categories, S, H, V, SH, and HR, plus three severe subcategories, S3, H2, and V2. 5 plus 3 is 8. 8 is not 13. The API's illicit and illicit/violent are not in that eight. The API splits self-harm into self-harm, self-harm/intent, and self-harm/instructions. The paper's SH is one label. One label is not three keys. The introduction says that among sampled user prompts only 0.04% of cases included self-harm. Table 4's random-sampling cell for SH is 0.09%. 0.04 is not 0.09. Do not replace the introduction with the table (Markov and colleagues, A Holistic Approach to Undesired Content Detection in the Real World).

Table 3 is AUPRC, and a star means the taxonomy match was approximate. On the public set, sexual is 0.8709 for Perspective and 0.9703 for theirs, and the Perspective cell is starred. Hate is 0.6914 and 0.7968, with no star on either. Violence is 0.5201 and 0.7371. Harassment is 0.3902 starred and 0.6191. 0.3902 is not 0.6191. Self-harm has no Perspective number, printed as a dash, and theirs is 0.8070. A dash is not 0. On Jigsaw, toxic is 0.9769 for Perspective and 0.9304 for theirs, and theirs is starred. Theirs is lower. The prose says Perspective does a better job on Jigsaw data. That sentence matches the toxic direction and does not match the public sexual direction, where theirs is higher. 0.9769 is not 0.9703. TweetEval hate is 0.5961 and 0.6473. Offensive is 0.7919 starred and 0.7024 starred. On offensive, theirs is lower. 0.7024 is not 0.6473. Stormfront hate is 0.8754 and 0.9053. Reddit sexual is 0.8961 starred and 0.9417 starred. None of these cells is 0.9999992735124786, and none of them is a boolean.

Table 4 is a sampling table, and the multipliers are not one number. Random versus active learning: sexual 1.49% and 25.53%, printed 17.1 times; hate 0.17% and 3.09%, printed 18.2 times; violence 0.48% and 9.92%, printed 20.7 times; harassment 0.55% and 6.41%, printed 11.7 times; self-harm 0.09% and 1.85%, printed 20.6 times; S3 0.24% and 2.42%, printed 10.1 times; H2 0.03% and 0.67%, printed 22.3 times; V2 0.25% and 4.27%, printed 17.1 times. Sexual and V2 are both printed as 17.1 times. 25.53 divided by 1.49 is 17.13, which rounds to 17.1. 4.27 divided by 0.25 is 17.08, which also rounds to 17.1. The quotients are not the same, and the printed multiplier is. Do not treat the shared 17.1 as one ratio. Safe is 96.57% of the random sample and 59.54% of the active-learning sample. The caption says one sample can carry multiple labels, so the percentages sum to more than 100%. The active-learning column, including safe, sums to 113.7. The random column sums to 99.87, which is not 100 and not 113.7. The prose says about 40% of active-learning samples trigger at least one undesired label, and only 3.4% of random samples do. 40 is not 3.4. 59.54 is the safe cell, not the 40. 96.57 is not 3.4. The procedure scores 500,000 production samples and then chooses about 2,000. 500,000 is not 2,000. The initial training set is 6,000 and the validation set is about 5,500. 6,000 is not 5,500. Iterations N equal 3. 3 is not 8 categories.

Table 5 is a third grid, and the sentence about 69,000 examples points at one pair of cells. PUB is around 90,000 public examples. SYN adds 69,000 synthetic examples. MIX adds 60,000 labeled production samples on top of SYN. 90,000 is not 69,000. 69,000 is not 60,000. Do not add them into a total the paper does not print. The prose says the average AUPRC on hateful content improved from 0.417 to 0.551 by adding those 69,000 examples. Table 5's hate row, baseline column, is 0.417 on PUB and 0.551 on SYN. That pair matches the sentence. The DAT cell on PUB hate is 0.491, and the DAT cell on SYN hate is 0.476. 0.491 is not 0.551. 0.476 is not 0.417. MIX hate is 0.843 baseline and 0.818 with DAT. 0.843 is not 0.551. Self-harm baseline moves from 0.063 on PUB to 0.281 with DAT, from 0.086 to 0.296 on SYN, and from 0.621 to 0.632 on MIX. 0.063 is not 0.621. 0.632 is not 0.8070 from Table 3. An input-reduction check says 17 of 21 training examples containing the phrase all immigrants were undesired and 4 were safe. 17 plus 4 is 21. Characters per sample fell from 722.3 to 15.9. 722.3 is not 15.9. About 97% of the key tokens were unsafe and the remaining 3% were not. 97 plus 3 is 100. Learning rate 0.05, batch size 256, and dropout 0.1 are training settings. 0.05 is not 0.1, and 256 is not a category score.

Do the three proofs on one real call, in the log that actually holds it. First, the binding. Copy the moderation id, the model, and every category you are willing to defend, with the boolean and the score. If the call was not stored, write that. Do not infer omni-moderation-latest because one example prints it, and do not infer that every boolean is true because the placeholder example is shaped that way. If a score is 0 and the boolean is true, write both. Do not repair the pair. Second, the other call. A different moderation id, with a different boolean or a different score on the category you named, is not this one. An AUPRC from a paper is not that second call. If you cannot show a call that is not this category result, you have a chart, not a record. Third, the stored response. The object you kept is the proof. A sentence the model produces when you ask it what it flagged is not the object. If the log does not keep the moderation id, write that.

Three moves feel like the proof and are a second problem. Turning the feature off because a score looked large, and then describing the missing id as a cleanup. Quoting 0.04% because it looks precise, while the call you were asked about is one id, and the table in the same paper prints 0.09% for a different cut of self-harm. Asking the model which category fired. The model will pick a plausible name from the thirteen. It will not open the stored object. A stop reason you have not kept, while the user is arguing about a cut-off answer, is name the stop the response carried. Opening that field does not name the category.

The path is ordinary when an operator can open one assistant's stored call without opening the next assistant's call as the way to prove it. Inflectiv Helios is a multi-assistant platform with prompt monitoring. The shape that matters here is one assistant's call you can point at, distinct from the next assistant's call. That case is not a moderation ledger. Nothing in the description is a headcount, a duration, or a result figure. If the only control on the screen shows a dashboard and will not name a moderation id, you do not have this proof. You have a chart.

There is a version of this afternoon where a rescue is the wrong purchase. Nothing is in production. No model is being called, so there is no moderation field to read. The request was not logged. A customer has already seen another customer's data. A category boolean does not close a leak. The wrong value was already a write. The field tells you what the classifier stored. It does not undo the row. Or no metric was ever named, so there is nothing a later eval can fail. The questions for that gap are not this page.

When the call is real and the log cannot name the category, the missing object is specific. The moderation id, the model, the category, the boolean, and the score, plus a second call that is not this one, plus the fact that you stored the response. An AI pilot failed to reach production engagement is for the system that is already in front of customers and cannot yet show those three: audit in the first one to two weeks, finished in 8 to 12 weeks, scoped after the audit, unpaid if the agreed metric does not move, rollout behind a human-review gate. The triple you wrote tonight is the kind of case the harness has to include. An AI integration sprint is the shape where the acceptance sentence can require the moderation object on every stored call that is supposed to be filtered. If you do not yet know whether the pilot can be saved, a two-week AI readiness assessment hands you the findings either way. Those are the terms on the site. They are not a promise that a paper's AUPRC applies, and they are not 0.9999992735124786.

The customer a charge names, and the workflow a run executed, are different objects: which customer did that charge name? and which workflow did that run execute?. Do not record either on this page. A customer id is not a category. A YAML path is not a score.

Before you tell them the filter cleared it, do the three proofs and write the category. Name the moderation id, the model, the boolean, and the score, or write that the call was not stored. Show a different call that is not that result. Show that the object came from the log and not from the model remembering. Do not treat a score of 0 as a pass when the boolean says true, and do not ask the model which of the thirteen it was. A paper can tell you how a different taxonomy scored on someone else's sets. The binding is the call you can defend.

Questions

You stop treating the reply as something a filter cleared, and you stop asking the model whether it was safe, until you can name the moderation id, the model, the category, the boolean, and the score, show a different call whose category or score is not this one, and show that you stored the response. If the log does not keep the call, write that.

No. An area under a precision-recall curve is a score on someone else's set. A category you can defend is the boolean and the score on this stored moderation object, or a written line that the call was not stored. 0.04% in a paper's introduction is not 0.09% in that paper's sampling table, and a score of 0 beside true is not a pass.

When nothing is in production, when no model is being called, when the request was not logged, when a customer has already seen another customer's data, when the wrong value was already a write, or when no metric was ever named.

Find this useful? Tell Google to show you more of it.

Let's put AI to work in your business.

A 30-minute call. You bring the workflow or the roadmap — we'll tell you what's feasible, what it costs, and what we'd build first.

Book a call→