Serotonin Receptor Subtypes Shape Biphasic Behavioral Responses to DOM: Mechanistic Insights from Mouse Models

Study Background and Research Question

Serotonergic hallucinogens such as (-)-2,5-dimethoxy-4-methylamphetamine (DOM) have long served as valuable tools in psychopharmacology, due to their ability to induce robust behavioral and perceptual changes in animal models and, by extension, in humans. Among these, the head twitch response (HTR) and alterations in locomotor activity are established readouts for serotonergic activity in rodents. Notably, DOM and related phenylalkylamine hallucinogens display an inverted U-shaped (biphasic) dose-response for both HTR and locomotion, a phenomenon whose underlying receptor mechanisms have remained incompletely resolved. This study, published in Psychopharmacology (2024), directly investigates how 5-HT2A, 5-HT2C, and 5-HT1A serotonin receptor subtypes contribute to these biphasic behavioral responses in C57BL/6J mice.

Key Innovation from the Reference Study

The principal innovation of this work lies in its systematic pharmacological dissection of the serotonergic receptor subtypes responsible for the DOM-induced biphasic dose-response in both HTR and locomotor activity. Prior studies had established that the 5-HT2A receptor is necessary for the characteristic effects of hallucinogens, but the distinct roles of 5-HT2C and 5-HT1A receptors—especially at different points of the dose-response curve—were poorly defined. By using selective receptor agonists and antagonists, as well as a Gαi/o inhibitor, the authors provide new mechanistic clarity and reveal that 5-HT2C and 5-HT1A receptors distinctly modulate the magnitude and direction of DOM’s behavioral effects, especially at higher doses where the U-shaped curve inverts.

Methods and Experimental Design Insights

The research team employed a robust pharmacological approach in adult C57BL/6J mice. Key interventions included:

• Administration of DOM across a range of doses (0.615–10 mg/kg, i.p.) to establish the biphasic response profiles for HTR and locomotion.
• Selective antagonism of 5-HT2A (M100907), 5-HT2C (SB242084), and 5-HT1A (WAY100635) receptors to parse their respective contributions.
• Agonism of 5-HT1A receptors (8-OH-DPAT) to probe suppressive regulatory effects.
• Use of pertussis toxin (PTX), a well-characterized AB5-type protein exotoxin and Gαi/o inhibitor, administered intracerebroventricularly to disrupt Gi/o-coupled receptor signaling and test for cAMP pathway involvement.

Behavioral endpoints were recorded as head twitch counts and total locomotor activity, with all protocols validated against established preclinical models for hallucinogen action.

Protocol Parameters

• DOM dosing: 0.615–10 mg/kg, intraperitoneal injection, administered in a volume and vehicle compatible with rodent behavioral assays.
• 5-HT2A antagonist (M100907): 500 μg/kg, i.p., used to fully block HTR; 50 μg/kg, i.p., to examine effects on low-dose DOM-induced locomotion.
• 5-HT2C antagonist (SB242084): 0.3–1 mg/kg, i.p.; 1 mg/kg dose notably increases locomotor activity at all DOM doses.
• 5-HT1A antagonist (WAY100635): 1 mg/kg, i.p.; enhances both HTR and locomotor activity.
• 5-HT1A agonist (8-OH-DPAT): 1 mg/kg, i.p.; suppresses HTR and locomotor activity.
• Pertussis toxin (PTX): 0.25 μg/mouse, intracerebroventricular (i.c.v.) pretreatment; enhances HTR and attenuates DOM’s effect on locomotion.

These parameters provide a foundation for future studies seeking to dissect serotonergic modulation of behavior using refined pharmacological tools.

Core Findings and Why They Matter

The major findings of the study can be summarized as follows (see reference):

• 5-HT2A receptor antagonism (M100907) fully abolished the head twitch response at all DOM doses and reduced low-dose DOM-induced locomotor hyperactivity, confirming its essential role in mediating classic hallucinogenic responses.
• 5-HT2C receptor antagonism (SB242084) reduced HTR only at an intermediate dose (2.5 mg/kg DOM) but increased locomotor activity across all DOM doses, suggesting a complex modulatory influence over behavioral outcomes.
• 5-HT1A receptor antagonism (WAY100635) paradoxically enhanced both HTR and locomotion, while 5-HT1A agonism (8-OH-DPAT) exerted the opposite effect, indicating that 5-HT1A activity provides negative feedback or dampening of behavioral excitation induced by DOM.
• Pertussis toxin (PTX) pretreatment amplified the HTR and reduced DOM’s locomotor effects, implicating Gi/o protein-mediated pathways—often linked to cAMP signaling—in the regulation of these biphasic responses. This aligns with the known ability of PTX to modulate cAMP-dependent signaling in the brain.

The crux of these findings is that the inverted U-shaped dose-response of both HTR and locomotor activity induced by DOM is not governed solely by the 5-HT2A receptor, but rather by a nuanced interplay between 5-HT2A, 5-HT2C, and 5-HT1A receptor subtypes, each influencing different behavioral endpoints and different regions of the dose-response curve.

Comparison with Existing Internal Articles

This study extends mechanistic insights previously discussed in "5-HT2C/1A Receptor Roles in DOM-Induced Biphasic Responses in Mice" by providing detailed pharmacological dissection and linking behavioral outcomes to specific receptor subtypes. The present findings complement research on immune and neuroimmune modulation, such as the role of AB5-type protein exotoxins in cAMP signaling, explored in "Pertussis Toxin: Mechanistic Insights and Advanced Research Horizons". Both lines of work underscore the value of immune response modulation and G protein-coupled receptor (GPCR) signaling analysis for unraveling complex neurobehavioral and immune mechanisms. While the current study focuses on serotonergic signaling in the CNS, the shared methodological use of pertussis toxin as a Gi/o inhibitor bridges neuropharmacology and immunology, as highlighted in "Pertussis Toxin: Applied Immune Modulation and Workflow Insights", where detailed protocols for immune cell signaling are discussed.

Limitations and Transferability

Despite its strengths, this study is constrained by several factors. First, the conclusions are drawn from acute pharmacological manipulations in mice, which may not fully translate to chronic or clinically relevant exposure scenarios. The behavioral paradigms, while validated, are proxies for human perception and psychiatric symptoms, not direct analogs. Additionally, the effects of DOM and receptor-specific drugs may differ across mouse strains, sexes, or developmental stages. The use of pertussis toxin, while effective for dissecting Gi/o-coupled signaling, carries caveats related to off-target effects and the challenge of translating toxin-based interventions to human therapeutics. Finally, although the study provides robust evidence for receptor involvement, it does not address downstream intracellular signaling cascades in detail, leaving open questions about the molecular effectors of the observed behavioral changes.

Research Support Resources

Researchers aiming to explore cAMP-dependent signaling or immune response modulation in CNS or immunological models can leverage high-purity Pertussis toxin (SKU B7273) as an effective Gαi/o inhibitor for both in vivo and ex vivo protocols. According to the product information, this AB5-type protein exotoxin is widely used for dissecting GPCR pathways and cAMP signaling in a variety of systems, including neuronal and immune cell assays. When using such reagents, it is recommended to prepare fresh solutions and adhere to validated dosing protocols for maximum activity and reproducibility. For additional protocol guidance and troubleshooting, see the workflow discussion in internal resources focused on immune modulation.