Python c+++

Processing in Python
How to take advantage of spaCy & a bit of Cython for blazing fast NLP

I also published a Jupyter notebook with the examples I describe in this post.
When we published our Python coreference resolution package✨ last year, we got an amazing feedback from the community and people started to use it for many applications 📚, some very different from our original dialog use-case 👥.
And we discovered that, while the speed was totally fine for dialog messages, it could be really slow 🐌 on larger news articles.
I decided to investigate this in details and the result is NeuralCoref v3.0 which is about 100 times faster 🚀 than the previous version (several thousands words per seconds) while retaining the same accuracy, and the easiness of use and eco-system of a Python library.
In this post I wanted to share a few lessons learned on this project, and in particular:
How you can design a high-speed module in Python,
How you can take advantage of spaCy’s internal data structures to efficiently design super fast NLP functions.
So I am a bit cheating here because we will be talking about Python, but also about some Cython magic — but, you know what? Cython is a superset of Python, so don’t let that scares you away!
Your current Python program is already a Cython program.
There are several cases where you may need such speed-ups, e.g.:
you are developing a production module for NLP using Python,
you are computing analytics on a large NLP dataset using Python,
you are pre-processing a large training set for a DeepLearning framework like pyTorch/TensorFlow, or you have a heavy processing logic in your DeepLearning batch loader that slows down your training.
One last thing before we start: I also published a Jupyter Notebook with the working examples I talk about in this post. Try it out!
First step to rocket speed: Profiling
raw array of C pointers but you can also choose other options, in particular C++ structures like vectors, pairs, queues and the like. In this snippet, I also used the convenient Pool() memory management object of cymem to avoid having to free the allocated C array manually. When Pool is garbage collected by Python, it automatically frees the memory we allocated using it.
A good reference on the practical usage of Cython in NLP is the Cython Conventions page of spaCy’s API.
👩‍🎨 Let’s Try that Code!
There are many ways you can test, compile and distribute Cython code! Cython can even be used directly in a Jupyter Notebook like Python.
First install Cython with pip install cython

The first thing to know is that most of your code is probably just fine in pure Python but there can be a few bottlenecks functions that will get you orders of magnitude faster if you give them some love.
You should thus start by profiling your Python code and find where the slow parts are located. One option is to use cProfile like that:

You’ll likely find that the slow parts are a few loops, and some Numpy arrays manipulations if you use neural networks (but I won’t spend time talking about NumPy here as there is already a lot of information written about that).
So, how can we speed up these loops?
Fast Loops in Python with a bit of Cython……
………..raw array of C pointers but you can also choose other options, in particular C++ structures like vectors, pairs, queues and the like. In this snippet, I also used the convenient Pool() memory management object of cymem to avoid having to free the allocated C array manually. When Pool is garbage collected by Python, it automatically frees the memory we allocated using it.
A good reference on the practical usage of Cython in NLP is the Cython Conventions page of spaCy’s API.
👩‍🎨 Let’s Try that Code!
There are many ways you can test, compile and distribute Cython code! Cython can even be used directly in a Jupyter Notebook like Python.
First install Cython with pip install cython
JHere is how the fast Cython version of our Python module looks like:

Here we used a raw array of C pointers but you can also choose other options, in particular C++ structures like vectors, pairs, queues and the like. In this snippet, I also used the convenient Pool() memory management object of cymem to avoid having to free the allocated C array manually. When Pool is garbage collected by Python, it automatically frees the memory we allocated using it.
A good reference on the practical usage of Cython in NLP is the Cython Conventions page of spaCy’s API.
👩‍🎨 Let’s Try that Code!
There are many ways you can test, compile and distribute Cython code! Cython can even be used directly in a Jupyter Notebook like Python.
First install Cython with pip install cython
J………4 second to get the answer. If we had a million documents it would take more than a day to give us the answer.
We could use multiprocessing but it’s often not such a great solution in Python because you have to deal with the GIL 😕 Also, note that Cython can also use multi-threading! And that may actually even be the best part of Cython because the GIL is released so we are at full speed 🏎 Cython basically directly call OpenMP under the hood. I won’t have time to talk about parallelism here so check this link for more details.
Now let’s try to speed up our Python code with spaCy and a bit of Cython.
First, we have to think about the data structure. We will need a C level array for the dataset, with pointers to each document’s TokenC array. We’ll also need to convert the test strings we use (“run” and “NN”) to 64-bit hashes.
When all the data required for our processing is in C level objects, we can then iterate at full C speed over the dataset.
Here is how this example can be written in Cython with spaCy:

On the left I wrote a script that builds a list of 10 documents parsed by spaCy, each with ~170k words. We could also have 170k documents with 10 words in each (like a dialog dataset) but that’s slower to create so let’s stick with 10 docs.
We want to perform some NLP task on this dataset. For example, we would like to count the number of times the word “run” is used as a noun in the dataset (i.e. tagged tagged with a “NN” Part-Of-Speech tag by spaCy).
A Python loop to do that is short and straightforward:

But it’s also quite slow! On my laptop this code takes about 1.4 second to get the answer. If we had a million documents it would take more than a day to give us the answer.
We could use multiprocessing but it’s often not such a great solution in Python because you have to deal with the GIL 😕 Also, note that Cython can also use multi-threading! And that may actually even be the best part of Cython because the GIL is released so we are at full speed 🏎 Cython basically directly call OpenMP under the hood. I won’t have time to talk about parallelism here so check this link for more details.
Now let’s try to speed up our Python code with spaCy and a bit of Cython.
First, we have to think about the data structure. We will need a C level array for the dataset, with pointers to each document’s TokenC array. We’ll also need to convert the test strings we use (“run” and “NN”) to 64-bit hashes.
When all the data required for our processing is in C level objects, we can then iterate at full C speed over the dataset.
Here is how this example can be written in Cython with spaCy:

The code is a bit longer because we have to declare and populate the C structures in main_nlp_fast before calling our Cython function [*].
But it is also a lot faster! In my Jupyter notebook, this Cython code takes about 20 milliseconds to run which is about 80 times faster than our pure Python loop.
The absolute speed is also impressive for a module written in a Jupyter Notebook cell and which can interface natively with other Python modules and functions: scanning ~1,7 million words in 20 ms means we are processing a whopping 80 millions words per seconds.
This concludes our quick introduction on using Cython for NLP. I hope you enjoyed it.
There are a lot of other things to says on Cython but it would get us too far from this simple introduction. The best place to start from now is probably the Cython tutorials for a general overview and spaCy’s Cython page for NLP.
Don’t hesitate to give us a few claps 👏 if you want more content like that!
*. ^ If you use low level structures several times in your code, a more elegant option than populating C structures each time, is to design our Python code around the low level structures with a Cython extension type wrapping the C level structures. This is how most of spaCy is structured and it is a very elegant way to combine fast speed, low memory use and the easiness of interfacing with external Python libraries and functions.
Thanks to Anthony MOI.

The code is a bit longer because we have to declare and populate the C structures in main_nlp_fast before calling our Cython function [*].
But it is also a lot faster! In my Jupyter notebook, this Cython code takes about 20 milliseconds to run which is about 80 times faster than our pure Python loop.
The absolute speed is also impressive for a module written in a Jupyter Notebook cell and which can interface natively with other Python modules and functions: scanning ~1,7 million words in 20 ms means we are processing a whopping 80 millions words per seconds.
This concludes our quick introduction on using Cython for NLP. I hope you enjoyed it.
There are a lot of other things to says on Cython but it would get us too far from this simple introduction. The best place to start from now is probably the Cython tutorials for a general overview and spaCy’s Cython page for NLP.
Don’t hesitate to give us a few claps 👏 if you want more content like that!
*. ^ If you use low level structures several times in your code, a more elegant option than populating C structures each time, is to design our Python code around the low level structures with a Cython extension type wrapping the C level structures. This is how most of spaCy is structured and it is a very elegant way to combine fast speed, low memory use and the easiness of interfacing with external Python libraries and functions.
Thanks to Anthony MOI.
ProgrammingNLPPythonChatbotsOptimisation

Advertisements

Cherries ..pitting

HOW TO
How to Pit Cherries Even If You Don’t Have a Cherry Pitter
Lauren Cahn
Photo of cherry berries in the wooden bowl on the wooden table and fruit pits near it Shutterstock / Artem Malov

If you think pitting cherries is the pits, you’re not alone. But we’ve tried every hack imaginable, and one stands out above all the others.

For fans of fresh, home-baked cherry pie, we’ve got some good news and some bad news. The good news is we’ve got an absolutely amazing recipe for cherry pie (and other fresh cherry recipes!) that will make the process of pitting cherries feel totally worth it. The bad news is that the process is long and arduous.

But does pitting cherries really have to be such an arduous process? The good news is no! You can pit cherries the easy way with or without the fancy pitter. Here’s how.

How to pit cherries with a standard pitter
You may not want to hear this, but cherry pitters were invented for a reason. The standard cherry pitter grips the cherry, pokes the pit through swiftly, smoothly and easily, and has some sort of “splash guard” (because poking a pit through a cherry involves placing pressure on the fleshy, red orb, which will cause its juices to squirt all over). You can purchase one here for under $15, and it’s a worthy investment if you just can’t get enough of cherries. And who can with these amazing recipes?

Bottom line: This is the gadget you need if you’re serious about your cherry treats.

How to use a mason jar pitter
But the traditional pitter isn’t the only game in town. There’s now an attachment for a standard mason jar you can use. This contraption allows you to place the cherry in the designated spot and use the plunger to pit the cherry. The convenient part is that the mason jar collects all those pits in one spot so you don’t have to chase them across the kitchen floor.

Bottom line: It’s a bit more expensive, but it adds a level of convenience.

How to pit cherries without any special equipment
The internet would have you believe you can pit cherries with all sorts of household do-dads from paperclips to straws. Some of these tricks are easier said than done. Here’s the lowdown on these techniques:

How to pit cherries with a paper clip: Unbend a paper clip, wash it, de-stem all your cherries, and use one end of the paper clip to poke into the cherry, stick firmly into the pit, and push the pit through. Or, don’t. Because this process requires agonizing exactness, is slow and extremely messy.
How to pit cherries with a straw: You can’t. The straw bends, and it just doesn’t work. That said, using a straw is a great way to hull strawberries with almost zero waste.
How to pit cherries with a chopstick or a pastry tip: Poking a pit through the firm flesh of a cherry using a chopstick is virtually impossible. The blunt edge doesn’t even bite into the cherry pit the way a paper clip does. The pastry tip is slightly better than the chopstick but less effective than the paper clip, I suppose because your fingers are closer to the cherry and therefore able to hit the pit with more accuracy.
Bottom line: While these are fun hacks to try, we can’t in good faith recommend them for anything but a little (messy) fun.

The easiest, no-tools-needed way to pit cherries
The fastest, easiest, least frustrating, least expensive way to pit cherries is to use your hands, or, more specifically, your fingers. Simply pull out the stem, and push your finger through the non-stem end. The cherry will break into two pieces, maybe three, and the pit will be free. Done.

Yes, if you’re using dark red cherries (as opposed to, say, Ranier cherries), your fingers will look like Lady MacBeth’s. Is it that terrible to get a little juice on your fingers when the end result is a heap of freshly pitted cherries you can use in this Michigan cherry pie?

Try these cherry recipes from Grandma’s recipe box.
Chocolate Cherry Layer Cake
Chocolate Cherry Layer Cake

Cherry-Berry Streusel Pie
Cherry-Berry Streusel Pie

Almond Cherry Cobbler
Almond Cherry Cobbler

Berry Cherry Pie
Berry Cherry Pie

Minister’s Delight
Minister’s Delight

Black Forest Chocolate Torte
Black Forest Chocolate Torte

Cherry Spice Cake
Cherry Spice Cake

Michigan Cherry Pie
Michigan Cherry Pie

Cherry Pecan Upside-Down Cake
Cherry Pecan Upside-Down Cake

Cherry-Peach Dumplings
Cherry-Peach Dumplings

Cherry-Almond Streusel Tart
Cherry-Almond Streusel Tart

Cherry Crumb Dessert
Cherry Crumb Dessert

Apple-Cherry Cream Cheese Pie
Apple-Cherry Cream Cheese Pie

Cherry Cream Pie
Cherry Cream Pie

Double Cherry Pie
Double Cherry Pie

Rhubarb-Cherry Pie
Taste of Home
Rhubarb Cherry Pie

Cherry Gelatin Supreme
Cherry Gelatin Supreme

Mid term Elections USA

What the Democrats Must Do Now to Defeat Trump
David Remnick

A Democratic majority in the House will not only thwart Donald Trump’s legislative ambitions; it could also intensify the state of crisis and siege in Washington.
Photograph by Aaron P. Bernstein / Getty
Donald Trump’s name was nowhere to be found on the ballots of the 2018 midterm elections, but the vote, as he put it at one rally, “is a referendum about me.” In recent weeks, as he appeared at one rally after another, the President became himself, only more so, slamming the press, sliming opponents, and waging the most bigoted national political campaign in this country since the days of George Wallace.

The Democratic Party failed to achieve a “blue wave,” an overwhelming victory that would have represented a nationwide repudiation of the 2016 election. Our divisions have only deepened. But Trump lost in some consequential ways. In a high-turnout election, the G.O.P. yielded control of the House of Representatives to the Democrats for the first time in eight years—a crucial check on Presidential power. A record number of women were elected to Congress—a reflection, in part, of a #MeToo movement that the President has disdained. At least four of them were young women of color, including Rashida Tlaib, in Michigan; Ilhan Omar, in Minnesota; Lauren Underwood, in Illinois; and Alexandria Ocasio-Cortez, in New York. Democrats also improved, since 2016, among Midwestern and suburban voters. And, for the second time in two years, Trump and his allies lost the popular vote.

The midterm results, however, will provide Trump adequate reason to claim victory and, despite his low approval ratings, feel confident that he can win a reëlection campaign that leans heavily on rural voters. Republicans continue to dominate beyond the cities and suburbs, and they widened their margin in the Senate, which is a far less representative body than the House. They also held off statewide Democratic challenges in Florida and Texas, though the narrow margins of victory in those red states should give the G.O.P. leadership pause. Beto O’Rourke, who narrowly lost in the Texas Senate race to the incumbent, Ted Cruz, ran an especially compelling race.

Trump will doubtless take no blame for his party’s loss of the House. He never takes blame for anything. In the last days of the campaign, he made sure to inoculate his political ego against criticism by saying that there just wasn’t enough of him to go around: he could not campaign for many House candidates—there were so many. He could even blame “illegal voters” for his shortfall in the popular vote, as he did in 2016.

A Democratic majority in the House will not only make it harder for Trump to achieve his legislative ambitions; it could also intensify the state of crisis and siege in Washington. The loss in the House of Representatives means that an array of committees—Judiciary, Intelligence, Ways and Means, Foreign Affairs, and others—will now be chaired by Democrats who can initiate, or accelerate, investigations into Trump’s past, his Presidency, and his associates. They replace Republican chairmen like Devin Nunes, who, as the chairman of the House Intelligence Committee, often seemed to act less as a detached lawmaker and more like the President’s personal attorney.

The Republican leadership in the House has been dreading this prospect for months. Late last summer, Axios reported that the G.O.P. had compiled a spreadsheet of the many areas in which Trump was vulnerable to investigation. The spreadsheet includes, Axios reported, “more than one hundred formal requests from House Democrats [in] this Congress, spanning nearly every committee.” Democratic-led committees could issue subpoenas to investigate the President on a range of subjects. The Ways and Means Committee can request his tax returns. Other committees can launch more aggressive inquiries into his communications and business relations with Russia; his family businesses; possible money-laundering schemes; his payment to Stormy Daniels; the firing of the former F.B.I. director James Comey and various U.S. attorneys; the Muslim travel ban; the family-separation policy on the southern border; the response to Hurricane Maria in Puerto Rico; Jared Kushner’s compliance with ethics laws; election security; Treasury Secretary Steven Mnuchin’s business dealings; and more.

Whether the House leadership pursues impeachment remains a vexed strategic question. The Republicans hold a decisive majority in the Senate, making a conviction nearly impossible. What’s more, Democratic lawmakers are wary of a precipitous move toward impeachment; they recall how the Bill Clinton impeachment was, in the end, a political debacle for the G.O.P. They also know that Trump has proved repeatedly that he is an impulsive and vengeful politician on an ordinary day; when he is cornered, when he feels under attack from the press and his opponents, there is little he will not do or say.

The last weeks of the campaign were full proof of that. Though Presidents, even seemingly popular Presidents, frequently lose ground in midterm elections—under Barack Obama, the Democrats lost sixty-three House seats in 2010—Trump made his historical mark in these midterms by running a campaign distinguished by its naked appeals to racism, xenophobia, and paranoia. These were not inadvertent gestures. They were not gaffes. He did this deliberately and incessantly. His calculation, as it had been in 2016, was that his supporters’ deepest anxieties are connected to the country’s changing demographics. He showed little interest in running on matters of policy. Health care, it turned out, was a losing issue for him. Instead, without restraint or shame, he whipped one crowd after another into a frenzy by waving the banner of fear, resentment, and white nationalism.

Trump is hardly the first Republican to use race to define a national election. Barry Goldwater, Richard Nixon, Ronald Reagan, George H. W. Bush, and George W. Bush each used racial dog whistles or campaign commercials and surrogates to exploit the racial currents that persist in American life. Trump, however, made no effort to conceal his intent, no effort to employ a Lee Atwater to do his dirty work.

He warned that criminal gangs from Central America and the “Middle Eastern” terrorists in their midst—“the worst scum in the world”—were heading north toward Texas and carrying out a full-scale “invasion of our country.” At a rally in Florida, Trump said, “A Democrat victory on Election Day would be a bright, flashing invitation to traffickers, smugglers, drug dealers, and gang members all over the world. Republicans believe our country should be a sanctuary for law-abiding Americans, not criminal aliens.”

Trump felt no compunction about using the U.S. military as a political prop for his midterms fear campaign. He ordered thousands of active-duty troops to the border to ward off a caravan that was, in fact, many hundreds of miles away. Fox News amplified the sense of encroachment and insecurity by suggesting that the destitute men, women, and children in the caravan were being underwritten by the financier and philanthropist George Soros—a blatant anti-Semitic trope—and might even spread deadly diseases throughout America, after leaping Trump’s “beautiful” barbed wire.

Trump did not stop there. He called for the support of whom he portrayed as the nation’s beleaguered defenders: “Where are the Bikers for Trump? Where are the police? Where are the military? Where’s ICE? Where’s the Border Patrol? No, we’ve taken a lot.”

Calling on powers that he does not have, Trump said that he would override the Fourteenth Amendment of the Constitution and ban “birthright citizenship” by executive order. He was undeterred by any criticism, and spoke in larger truths. As he told supporters in Montana, “I’m the only one that tells you the facts.”

During the campaign, Trump seemed to eclipse George Wallace in his willingness to send blunt racial signals to his base voters. In 1963, Wallace, in his inaugural speech as the governor of Alabama, did not conceal his racism, saying that he was devoted to a policy of “segregation now, segregation tomorrow, segregation forever,” and would go to any lengths to protect what he called “the great Anglo-Saxon Southland.” But, in 1972 and 1976, during Wallace’s later runs for President, his language was somewhat more guarded. Evidently, Trump saw no reason for such restraint or euphemism.

The signal from the President to his party and its candidates was clear: do what is necessary. Often enough, they did. The ugliness reached its nadir on the eve of Election Day, when anti-Semites put out robocalls smearing Oprah Winfrey, who had campaigned in Georgia for the Democratic candidate for governor, Stacey Abrams. “This is the magical Negro Oprah Winfrey asking you to make my fellow Negress Stacey Abrams the governor of Georgia,” the voice on the calls said. “Years ago, the Jews who own the American media saw something in me—the ability to trick dumb white women into thinking I was like them and to do, read, and think what I told them to. I see that same potential in Stacey Abrams.”

There were some who called out Trump effectively, if not decisively. Last week, just days before the midterm elections, the Reverend Dr. William Barber, the pastor of the Greenleaf Christian Church in Goldsboro, North Carolina, and one of this country’s most powerful moral voices, went to a pulpit in Greensboro carrying a shofar, the ram’s horn that is sounded in synagogues on the High Holy Days. Barber, a hulking man who suffers from a painful affliction of the spine and joints, winced as he rose from his chair and then blew the shofar, summoning the crowd from song to contemplation of the historical moment.

“There are seasons when we are made to be still,” he said gravely. “And we need the kind of singing you just experienced, so that we can handle the nightmares.” But now there would be no more singing.

For weeks, Barber had been speaking at demonstrations and marches throughout the South, insisting that the states protect voting rights, particularly for people of color. He denounced the rhetoric of bigotry, misogyny, and conspiracy that has emanated from the White House for the past two years. But now, he said, he was at a loss. In late October, the United States had approached what Barber called “the precipice”—a moment of multiple pipe bombs and mass murder.

“Only the inability to detonate kept us from seeing possibly two former Presidents and their First Ladies, Attorney General, a [former] Vice-President . . . two sitting African-American senators, two philanthropic entrepreneurs, and an entire newsroom from being killed in one swift fell swoop,” Barber said.

Barber insisted that the pipe bombs mailed around the country represented a far greater peril than a pile of duds mailed by a maniac. “Do we realize,” he said, “what that could have done to the stability of the world?”

And then Barber spoke of the incident in Jeffersontown, Kentucky, in which Gregory Bush, an armed white man in his early fifties, banged furiously on the door of a predominantly black Baptist church; had the door not been locked, he might have repeated the kind of mass killing carried out in 2015, by a young white supremacist named Dylann Roof, at Emanuel A.M.E. Church, in Charleston, South Carolina. Bush, who had a record of mental illness and domestic abuse, got in his car and headed to a nearby Kroger supermarket, where he shot and killed two African-Americans, Vickie Lee Jones and Maurice Stallard.

Finally, the Reverend Barber came to the next horror, the slaughter in Pittsburgh. “Has it caused you to almost lose your mind to think that they were shot in a synagogue named Tree of Life?” he asked. The murders of eleven Jews at prayer, he said, represented a “lynching” by assault weapon. “I don’t know where this nation’s mind is,” Barber said. “It is not just a President. What is wrong with the crowds that cheer and cheer? What’s wrong with the politicians that turn the other way?”

For voters still traumatized by the shock of Trump’s ascent to the Presidency, Election Day was a source of anxiety. Two years ago, I was interviewing Obama in the Oval Office, just a few days after the 2016 elections. Trump had just visited Obama at the White House. The atmosphere throughout the building was funereal. Obama and his aides were stunned by Trump’s wandering attention span, his disinterest in the details of foreign and domestic policy. I had asked Obama about Election Day, what it was like when he was running for state senator, U.S. senator, and President, and, finally, as a surrogate for the person he hoped to be his Democratic successor. Even in his exhaustion and bewilderment, Obama painted a picture of voting that was half hopeful.

“I love the stillness and the mystery of the day or two before elections, because in a lot of ways everything goes radio-silent,” he said. “Nobody at that point is really listening to an argument. The infrastructure is set. And now it’s this weird alchemy that’s taking place in the country, and you just have to kind of wait and see how it works. But there’s always this mystery to it, this possibility.”

“Which, in some ways, is powerful and affirming of the humanity of democracy, right?” he continued. “It’s not mechanical. It’s not a formula. It’s not set. It’s not fixed. There is always the possibility of surprise. And in that sense it’s a little bit like sports. It doesn’t matter what the odds are. Weird stuff happens. And that makes it scary if you’re rooting for one team or the other, but that’s the drama of it.”

Obama had originally hoped, of course, to be succeeded by Hillary Clinton. Then he hoped that Trump, once he took office, would become less mercurial, less theatrical, less mendacious, as he came to recognize the seriousness of his office. That, of course, never happened. It’s long been clear that Trump was never a mystery. He is not enigmatic. He is exactly who he seems to be. He ran as a nationalist, as a bigot, as an enemy of constitutional and global norms, and, from his first days in office, that is how he has governed.

Trump has become the putative leader of an international movement toward illiberalism, nationalism, protectionism, and xenophobia. He has proved an ally and an inspiration to autocrats from Brazil to the Philippines, supplying them with a vocabulary (“fake news”); a blunt, bullying style; and a sense of license. Those autocrats, who once might have regarded an American President with anxiety, as a brake on their most heedless impulses, are emboldened by Trump’s successes.

In domestic politics, Trump has transformed the Republican Party in his image. The G.O.P. leadership, which had once dismissed him as a buffoonish con and then as a danger, has capitulated to him almost entirely. The few Party leaders who have not become champions of the President have either retired or have taken extreme care to avoid criticism or confrontation.

And so the Trump emergency has hardly subsided. The powers of the Presidency are undiminished, and the Senate remains in Republican hands. His capacity to wreak havoc on constitutional and international norms persists. His capacity to pump toxic racism into the national atmosphere persists. His capacity to undermine truth itself persists.

The midterm elections have ended in a mixed result. The vote certainly was not a decisive repudiation of Trump, nor was it anything like the resounding endorsement he craved.

The enormous role played by women, both as candidates and as voters, is historic and promises more. Some of the losing statewide candidates—O’Rourke, in Texas, Andrew Gillum, in Florida—will surely be heard from again. Obama finally seems to have entered the fray without restraint. And the Democratic House, combined with whatever is to come from the Robert Mueller investigation, will put a keener focus on any crimes and misdemeanors. But in order to defeat him, defeat him decisively, the leadership of the Democratic Party will have to get better, become more focussed—and not merely on the travesty of its outrageous and dangerous opponent.

David Remnick has been editor of The New Yorker since 1998 and a staff writer since 1992. He is the author of “The Bridge: The Life and Rise of Barack Obama.”