Arrested for bigamy, broken headlight and inadequate windshield wipers. Does the NYPD know this data is public?

TLDR: On https://nyponline.org you can change the url parameters and receive things like disceplenary entries, arrests made with dates for entire career and maybe more. I made a twitter handle https://twitter.com/NYPDdata to tweet out the NYPD arrests made.

Reminder, NYPD officers are not your neighbors, thats a problem.

Intro

I was reading the ProPublic article on the NYPD (https://projects.propublica.org/nypd-disciplinary-records/) and saw this link to https://nypdonline.org/link/1026 a nypd website.

Its pretty cool, NYPD is providing a good amount of info on officers like rank, command history, training summary and other items.

You can see that for each tab the query is a api endpoint Request URL: https://oip.nypdonline.org/api/reports/13/datasource/list

I iterated through all the filter lists, here are some interesting items that don’t show up in the UI. It’s unclear if the NYPD knows that this data is also public.

Since 2018 NYPD has been publishing arrest data quarterly but that data does not include information on the arresting officer https://data.cityofnewyork.us/Public-Safety/NYPD-Arrest-Data-Year-to-Date-/uip8-fykc so there seems to be some new data here.

Examples for a officer LINK TO EXAMPLE FOLDER

Filters:

LIST_OF_FILTERS = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,1027,1028,1029,1030,1031,1032,1033,1034,1035,1036,1037,2041,2042]

1. Basic Profile Info, Ethnicity, Shield No,
2. Some kind of summary of command value and unspecified value.
3. List of arrests per year
4. Data for some kind of visualization involving arrests and misdemeanor information
5. Data for some kind of visualization, unclear
6. Another list of arrests per year
7. Some unknown value for command, PERMANENT is included in each json value
8. Says ArrestYear but a single value
9. List of all arrests the officer has ever made, just date and type of arrest
10. Some kind of history of a value for the officer, the values start with a PROB like 22 MONTHS PROB and then end with ANNUAL
11. Another visualization of ArrestYear
12. Unknown value of NO or YES with a date
13. Awards for EXCELLENT POLICE DUTY
14. Similar to 13
15. Another @ArrestYear but with different values. These seem to be some search of different items on Arrest per year. Perhaps type of arrest or characteristics of arrest.
16. Similar to above
17. Similar to above, all with different values. Must be some kind of filter on Arrests
18. Static image
19. Just name of officer and image
20. Value PERMANENT with date
21. When they were promoted to current command
22. Some record of officer discipline perhaps? Items like INTERNAL INVESTIGATION, CHRONIC SICK-A, DEPT. AUTO ACCIDENT,
23. Promotion history
24. Really long list of trainings: CUSTOMER SERVICE PROGRAM: ROLL CALL MODULE 1, INTRODUCTION TO PRECINCT GREETER PROGRAM , INVESTIGATIVE ENCOUNTERS-VIDEO #4 DOCUMENTATION AND SUPERVISION-UNIT TRAINING JULY 2016, COBRA REFRESHER TRAINING (THREAT)
25. List of events or trainings attended:POLICING A DEMOCRACY JULY 03. JUNE 07
26. List of arrest types but all with the same value, not sure what this is
27. “Search Instructions go here”, “Interactions”: [], “RelatedItems”:
28. More detailed courses:DETECTIVE BUREAU RETRIEVING BODY WORN CAMERA DATA, COUNTERTERRORISM U.A.S. (UNMANNED AIRCRAFT SYSTEMS) DRONE AWARENESS,IPHONE AND DAS MOBILE OPERATION, LOCATING APPLE DEVICES USING YOUR DEPARTMENT SMARTPHONE
29. Another “@ArrestYear” with different values
30. Unfilled Event details
31. Related to FOIL requests? List of values “FOIL 4234” with dates
32. FOIL & Discipline Documents is the key but no values
33. Some graph that is common across all officers: Lists things like FEMALE (6,191), MALE (26,539), NON-BI (1)
34. Some graph based on ethnicity same across all officers: Lists things like Asian 3055, BLACK 4933, NATIVE AMERICAN 23
35. empty
36. Active Disciplinary Cases = 100, same across all officers
37. VIOLATION, MISDEMEANOR, FELONY with values

CSV of Arrests

I parsed through all officers and saved all raw requests for each filter. You can see the data HERE . I also parsed all the arrest data into a single CSV which you can see HERE ( caution 5 Million rows, you will have to re-append the csv files together)

For CSV, this is the headers. There is data from way back in 1982.

Epoch_time_of_arrest,string_date_of_arrest,day_of_week,holiday,arrest_type,officer_name_without_comma,officer_tax_id,officer_race,officer_appointment,
officer_tenure,officer_assignment,officer_command,officer_rank,officer_shield,officer_substantiated_complaints,officer_unsubstantiated_complaints

Example data

378709200, 1/1/1982 12:00:00 AM, Friday, New Year’s Day, ROB-1ST:FORC THEFT/DEADLY WEAP, SOHAIB RAFI, 942750, ASIAN, 7/10/2006 12:00:00 AM, 14, 2/18/2021 12:00:00 AM, UNIFORMED OPERATIONS UNIT, DETECTIVE 2ND GRADE, 1044, 0, 0

378709200, 1/1/1982 12:00:00 AM, Friday, New Year’s Day, ROB-1ST:FORC THEFT/DEADLY WEAP, SOHAIB RAFI, 942750, ASIAN, 7/10/2006 12:00:00 AM, 14, 2/18/2021 12:00:00 AM, UNIFORMED OPERATIONS UNIT, DETECTIVE 2ND GRADE, 1044, 0, 0

Complaints are from the CCRB data.

Arrest Types

Some interesting items were the list of arrests by type and frequency. See HERE . There are 2100 things people were arrested for!!

The top

395723 CRIM POSS MARIHUANA-5TH:PUBLIC
381779 ASLT W/INT CAUSES PHYS INJURY
287489 INTENT/FRAUD OBT TRANS W/O PAY
279440 CRIM POSS CONTRL SUBST-7TH
246144 PETIT LARCENY
200643 VIOL OF LOCAL LAW VIOL
149816 CRIM POSSESSION STOLN PROP-5TH
145830 AGGRAVATED UNLIC OPER/MV-3RD
131656 AGGRAVATED UNLIC OPER VEH-3RD
109170 CPCS-3RD:NARC DRUG INT/SELL

WOW. Marijuana. Jumping turnstiles!!!

Jumping the turnstile is the third most common.

Ones I found interesting

13676 TRADEMARK COUNTERFEITING 3RD
8458 US CODE UNCLASSIFIED
7214 PROMOTING GAMBLING-2ND
6379 UNAUTHORIZED SALE OF FARECARD
6125 POSS/SELL UNSTAMPED CIGARETTES
5504 PUBLIC LEWDNESS
5222 MV LICENSE VIOL:NO LICENSE
5097 UNLAWFUL ASSEMBLY
2637 FORGERY-3RD
2252 UNLICENSED BOTTLE CLUB
2146 URINATE IN PUBLIC
1644 AUTO STRIPPING-3RD
1531 EQUIP VIO:WINDSHIELD TINT VIOL
1487 CPCS-4TH:HALLUCINOGEN
1435 UNAUTH PRACT PROFESSION
999 RIOT-2ND
864 FAILURE TO COMPLY WITH SIGN
840 POS GAMB DEVICE:SLOT MACHINE
689 INADEQUATE OR NO STOP LAMPS
643 INCITING TO RIOT
558 REFUSAL TO TAKE BREATH TEST
558 EQUIP VIO:HEADLIGHTS
495 JOSTLING HAND NEAR POCKET
400 BICYCLE OPER ON SIDEWALK
330 LOITERING
240 SEAT BELT VIOLATION-DRIVER
235 BICYCLE:FAIL TO KEEP RIGHT
202 UNAUTH RECORDING PERFORMCE-2ND
158 MFG DANGEROUS INSTRUMENT
85 SELL TIX W/O PRINTED PRICE
80 BRIBERY PUBLIC SERVANT-2ND
78 DISRUPT REL SER/FUNRL/MEMORIAL
64 BRIBING A WITNESS
48 ILL SIGNAL:LESS THAN 100 FEET
35 EAVESDROPPING
29 UNAUTH BOTTLING OF ALC BEV
29 CEMETERY DESECRATION-1ST DEG
24 TRANSPORT UNAUTH RECORDING-2ND
14 FORTUNE TELLING
10 LOG BOOK VIOLATIONS
10 INADEQUATE WINDSHIELD WIPERS
8 OPER PRIV MV W/O US MAIL SIGN
8 CONSPIRACY TO FORM MONOPOLY
7 TAKING BRIBE FOR PUBLIC OFFICE
7 ON PREMS RECORDKEEPING VIOL
6 LITTER RAILROAD RIGHT-OF-WAY
6 ABORTION 2ND
5 ILL OP SNOMOB
4 OWNING/HARBORING UNLICEN DOG
3 RADIO/TV/MOVIES PROCEEDINGS
2 TRUSTEE VOTE
2 FAIL TO CHANGE ADDRESS ON LIC
2 EQUIP VIO:SPLASH GUARDS
2 EQUIP VIO:NO DOOR HANDLES
2 ABORTION-1ST
1 UNLAW SALE BICYCLES
1 SNOWMOBILE - IMPRUDENT SPEED
1 KEEPING UNVACCINATED DOG
1 FAIL TO MAINTAIN PROPER FILES
1 BIGAMY
1 ANARCHY-PUB/DIST DOCUMENT

These are listed as arrests. But does this really mean someone was arrested for not having door handles? Or not wearing a seatbelt? Why are these arrests and not tickets? There are millions of tickets given each year in NYC, so these cant be tickets.

I tried cross referencing with the posted arrest data, for instance on 3/16/2021 in this data there is a arrest for Windshield Tint Violation. In the official NYPD quarterly releases, there is a 3/16/2021 arrest for TRAFFIC,UNCLASSIFIED INFRACTIO. ‘TRAFFIC,UNCLASSIFIED INFRACTIO’ never appears in the nypdonline dataset. So is someone doing some data ‘cleaning’? Why remove the data about tinting?

Twitter Account

Posts everyday https://twitter.com/NYPDdata the arrests from 2 weeks ago. I could move it to arrests from yesterday, but I think sometimes arrests are added later.

I could add more info like about how it compares historically etc.

Some Interesting Graphs

Graphs about arrest type

What about arrests by weekday for type of arrest

What about individual officers arrests?

Diving in, says that on 04/25/2018, tax id 929600 (https://oip.nypdonline.org/view/1/@TAXID=929600/) arrested 196 people. Seems like this officer is part of the special fraud squad and arrested them for GRAND LARCENY 3RD DEGREE.

Tenure and arrest frequency

Next Ideas

• FOIL for specifics about some arrets, like was someone really arrested for bigamy? Whats that story
• What is the variance of arrest per officer? Like do they only arrest for 1 type
• The CCRB compliant data is interesting, can we say something about the officers that have complaints on certain arrests by date.
• Can we cross reference this with the quarterly published arrests that are about the person arrested but not the arrested officer? To get a more complete dataset?
• Whats going on with some of these filters like filter 22 or 1027. There is some interesting data there.
• Is there grouping of arrest types by precinct and time? i.e. the spread of a arrest type over time from one location through city as police learn they can arrest for that, quota, or are instructed to?

Code

None polished. All hacked together with bugs. Github

comments powered by Disqus