From Serotonin to Triptans: Recognition of the Indole–Ethylamine Pharmacophore, C5 Substitution, and 5 HT₁B/5 HT₂B Selectivity

Introduction

Serotonin (5-hydroxytryptamine, 5HT) has a typical tryptamine structure: an ethylamine side chain is attached at the C3 position of the indole ring, and a hydroxyl group is present at C5. The antimigraine drug sumatriptan retains this indole–ethylamine scaffold while extending the C5 position with a bulkier sulfonamide-containing substituent.

Early radioligand-binding studies showed that sumatriptan had Kᵢ values of 17 and 27 nM for 5HT₁D and 5HT₁B binding sites, respectively, and a Kᵢ of 100 nM for 5HT₁A, whereas its Kᵢ values for another 10 neurotransmitter receptor-binding sites tested were all greater than 10 μM.[1] The principal receptor pharmacology of triptans is centered on 5HT₁B/5HT₁D. In the functional assays conducted by Wang et al., the tested triptans generally displayed markedly weaker activity at 5HT₂B than at 5HT₁B.[2]

Because 5HT₂B receptor agonism is associated with the risk of drug-induced cardiac valvulopathy, comparison of 5HT₁B and 5HT₂B provides a structural contrast between one of the principal therapeutic receptors of triptans and an important off-target receptor. Although 5HT₁D also contributes to the antimigraine pharmacology of triptans, it is outside the scope of this paired structural comparison.[2,3]

The crystal structures of human 5HT₁B and 5HT₂B receptors published in 2013 revealed that the two receptors possess similar deep orthosteric binding pockets but differ markedly in the geometry of the extended binding pocket near the extracellular side. The tryptamine-like core primarily occupies the conserved orthosteric site, whereas the C5 substituents of triptans extend toward the more divergent outer region. Together, these two components determine receptor recognition and subtype selectivity.[2,3]

1. Recognition of the Indole–Ethylamine Pharmacophore in the Orthosteric Pocket

1.1 The Protonated Amine Forms a Deep Ionic Anchor with Asp³·³²

5HT₁B is a class A G protein-coupled receptor (GPCR), and its ligand-binding cavity is located within the helical bundle formed by seven transmembrane helices. Asp129³·³² in 5HT₁B is located deep within the orthosteric binding pocket, where 3.32 denotes the generic Ballesteros–Weinstein residue number.

The crystal structure of 5HT₁B bound to ergotamine (ERG) shows that the positively charged nitrogen of the ligand forms a salt bridge with the carboxylate group of Asp129³·³². Molecular docking of 5HT based on the crystal structure reveals the same recognition mode, with the terminal ethylamine group, which is predominantly protonated under physiological conditions, oriented toward Asp³·³².[2]

This ionic interaction anchors the tryptamine side chain deep within the binding cavity and constrains the position and orientation of the indole ring in the orthosteric pocket.

1.2 Indole N–H and Thr³·³⁷ Provide a Second Orienting Interaction

In the 5HT₁B/ERG crystal structure, the indole N–H forms a hydrogen bond with Thr134³·³⁷; this interaction is also retained in the docking model of 5HT. Mutation of the corresponding Thr³·³⁷ residue to Ala in both 5HT₁B and 5HT₂B reduced the binding affinity of 5HT by more than tenfold, supporting the involvement of this site in 5HT recognition.[2]

The ionic anchoring of the protonated amine by Asp³·³² and the orienting interaction between Thr³·³⁷ and the indole N–H impose two spatial constraints, directing the indole–ethylamine scaffold into the receptor’s orthosteric site in a relatively well-defined conformation.

1.3 The Aromatic Surface of the Indole Ring Faces Transmembrane Helix V

In the docking model of 5HT, the indole ring faces transmembrane helix V. Nearby residues include Ser212⁵·⁴² and Ala216⁵·⁴⁶ in 5HT₁B, with Gly221⁵·⁴² and Ala225⁵·⁴⁶ occupying the corresponding positions in 5HT₂B. This interface is less polar than the corresponding region in several other biogenic-amine GPCRs and is compatible with the aromatic indole system of 5HT.[2]

The spatial arrangement formed by the protonated amine, the indole N–H, and the aromatic surface of the indole ring constitutes the fundamental recognition framework for tryptamine ligands.

2. The Conserved Orthosteric Pocket Provides Common Recognition, Whereas the Extended Pocket Underlies Subtype Differences

Sequence comparisons across the human 5HT GPCR family show that residues in the orthosteric binding pocket are substantially more conserved than those in the extended binding pocket. Superposition of the 5HT₁B/ERG and 5HT₂B/ERG crystal structures likewise shows that the overall geometry and key interactions of the deep orthosteric sites are highly similar in the two receptors.[2]

This conservation corresponds to the recognition of the shared endogenous ligand 5HT. Asp³·³², Thr³·³⁷, and the hydrophobic region surrounding the aromatic core maintain similar spatial relationships among different 5HT receptor subtypes, allowing a tryptamine-like core to establish a similar fundamental binding mode in multiple receptor subtypes.

The extended binding pocket near the extracellular entrance is less conserved. As a ligand extends from the orthosteric site in this direction, it contacts helical regions and amino acid side chains that differ substantially among receptor subtypes. The C5 substituents of triptans are positioned in this structural transition region.[2]

3. The Extended Pockets of 5HT₁B and 5HT₂B Differ Markedly in Geometry

3.1 The Extracellular End of Transmembrane Helix V Is More Open in 5HT₁B

Comparison of the 5HT₁B/ERG and 5HT₂B/ERG crystal structures shows that the extracellular end of transmembrane helix V in 5HT₁B is displaced outward by approximately 3.0 Å relative to that in 5HT₂B. This distance was measured from the Cα atom positions of Thr209⁵·³⁹ in 5HT₁B and Met218⁵·³⁹ in 5HT₂B.[2]

This displacement occurs above the orthosteric pocket, making the binding space near the extracellular entrance of 5HT₁B appreciably wider than that of 5HT₂B.

3.2 Thr209⁵·³⁹ and Met218⁵·³⁹ Further Alter the Available Space

The two receptors also differ in side-chain volume at position 5.39:

5HT₁B: Thr209⁵·³⁹

5HT₂B: Met218⁵·³⁹

The Thr side chain in 5HT₁B is relatively small, whereas the Met side chain in 5HT₂B is bulkier and extends farther into the ligand-binding region. The positional difference in transmembrane helix V, together with the difference in residue size at position 5.39, produces a more open extended pocket in 5HT₁B and a narrower extended pocket in 5HT₂B.[2]

These differences lie within the spatial range accessible to substituents at the C5 position of the indole ring and provide a structural basis for subtype recognition by triptans.

Molecular dynamics simulations further showed local differences in ligand–receptor contacts for ergotamine bound to the 5HT₁B and 5HT₂B receptors. In 5HT₁B, W⁶·⁴⁸ contacted ergotamine more frequently, whereas in 5HT₂B, ergotamine showed increased contacts with transmembrane helix V, including M⁵·³⁹.[5]

Figure 1. Differential Contacts of Ergotamine in the Binding Pockets of the 5HT₁B and 5HT₂B Receptors.

The upper panel shows the principal residues contacting ergotamine in the binding pockets of the 5HT₁B and 5HT₂B receptors; the lower panel shows the contact frequencies between ergotamine and the corresponding residues during molecular dynamics simulations. Purple indicates ligand–receptor contacts that differ between the two receptors. In 5HT₁B, W⁶·⁴⁸ contacted ergotamine more frequently, whereas in 5HT₂B, ergotamine showed increased contacts with transmembrane helix V, including M⁵·³⁹.[5] Source: Martí-Solano et al., PLOS ONE, 2014, Figure S1, CC BY 4.0.

4. C5 Substitution Translates Differences in Pocket Geometry into Triptan Selectivity

4.1 The C5 Substituent of Sumatriptan Extends into the Region Near Transmembrane Helix V

At the C5 position, 5HT bears only a hydroxyl group, whereas sumatriptan contains a bulkier sulfonamide-containing structure at the corresponding position. Its indole–ethylamine core can still enter the deep orthosteric pocket, while the C5-substituted portion extends toward transmembrane helix V and the extended binding pocket.

Molecular docking based on the crystal structures of the 5HT₁B and 5HT₂B receptors showed that sumatriptan can be accommodated relatively well within the 5HT₁B binding cavity. In 5HT₂B, steric hindrance caused by Met218⁵·³⁹ forces the sulfonamide group to adopt a different orientation and is accompanied by displacement of the indole core.[2]

The related 2013 studies obtained experimental structures of the 5HT₁B/ERG, 5HT₁B/DHE, and 5HT₂B/ERG complexes.[2,3] The specific binding poses of sumatriptan in the two receptors were derived from molecular docking based on these structures, with functional assays of triptans and receptor-mutagenesis results providing additional support for this spatial model.[2]

4.2 Mutation of Met218⁵·³⁹ Validates the Effect of Extended-Pocket Space on Triptan Potency

In functional assays of multiple triptans, Wang et al. found that donitriptan and eletriptan displayed relatively high potency in Gq protein-mediated signaling at the 5HT₂B receptor. Both compounds have relatively long and flexible linker regions at the C5 position of the indole ring, allowing conformational adjustment to accommodate the narrower extended binding pocket of 5HT₂B.[2]

When Met218⁵·³⁹ in the 5HT₂B receptor was mutated to Ala, the side-chain volume at this position decreased and the available space in the binding pocket increased. The functional potencies of both donitriptan and eletriptan at the 5HT₂B receptor increased substantially, although they remained lower than their potencies at the 5HT₁B receptor.[2]

These results indicate that the capacity of the region around position 5.39 to accommodate a ligand is one of the structural factors influencing the subtype selectivity of triptans. The bulky Met218⁵·³⁹ restricts favorable conformations of certain triptans in the 5HT₂B extended binding pocket, whereas reducing the side-chain volume at this position partially relieves this spatial constraint.[2]

5. The Effects of C5 Substitution Are Jointly Influenced by Volume, Extension Distance, and Conformational Flexibility

The effects of C5 substitution in triptans cannot be understood simply as “the larger the substituent, the greater the 5HT₁B selectivity.” Once the indole–ethylamine core has been directionally anchored by the orthosteric pocket, the volume of the C5 substituent affects its spatial occupancy near position 5.39, the length of the linker determines which region of the extended pocket the terminal group can reach, and conformational flexibility determines whether the substituent can reorient to accommodate local spatial constraints.[2]

Conformational flexibility may have a dual effect: it can help a ligand adapt to the extended pocket of the target receptor, but it may also increase its ability to adapt to an off-target receptor. Donitriptan and eletriptan have relatively long and flexible linker regions and retain comparatively high functional potency at 5HT₂B. The further increase in their potency following the M218⁵·³⁹A mutation also indicates that local space is closely related to conformational accommodation of the substituent.[2] Optimization at the C5 position must therefore exploit favorable accommodation in 5HT₁B while also assessing whether the length and flexibility of the linker increase the ligand’s ability to adapt to the extended pocket of 5HT₂B.

6. Paired 5HT₁B/5HT₂B Structures Support Coordinated Target–Off-Target Screening

In 2014, Rodríguez et al. further used the crystal structures of 5HT₁B and 5HT₂B for structure-based virtual screening. Approximately 1.3 million compounds were docked separately to the two receptors, and their predicted binding modes and scores were compared concurrently. Twenty-two candidate molecules were selected for experimental testing, of which 11 bound to 5HT₁B and nine showed more than fivefold selectivity for 5HT₁B over 5HT₂B. The most selective compound achieved approximately 306-fold selectivity.[4]

The same study also compared receptor crystal structures with high-quality homology models. Although the homology models used were generally highly accurate, they did not adequately reproduce the subtle structural features responsible for 5HT₁B/5HT₂B selectivity and therefore could not match the performance of the crystal structures in predicting selectivity.[4]

These findings and the structure–activity relationships of C5 substitution in triptans point to the same design principle: favorable binding to the target receptor and unfavorable spatial matching in the off-target receptor can be used simultaneously to establish selectivity. For highly homologous GPCR subtypes, helix displacements of only a few angstroms and differences in the side-chain volume of a single residue can alter ligand accommodation, binding conformation, and pharmacological profile.

7. Structure–Selectivity Relationships from Serotonin to Triptans

Ligand recognition by 5HT₁B and 5HT₂B can be divided into two interconnected structural levels.

In the deep orthosteric pocket, the protonated amine forms an ionic anchor with Asp³·³², the indole N–H establishes an orienting hydrogen bond with Thr³·³⁷, and the aromatic indole system matches the relatively nonpolar region near transmembrane helix V. These highly conserved interactions maintain the common receptor-recognition mode of tryptamine ligands.[2]

The extended binding region near the extracellular side differs more substantially between the receptor subtypes. The extracellular end of transmembrane helix V in 5HT₁B is displaced outward by approximately 3 Å and contains the smaller Thr209⁵·³⁹ residue, whereas the corresponding position in 5HT₂B is occupied by the bulkier Met218⁵·³⁹. When the C5 substituents of triptans extend into this nonconserved region, the subtle structural differences between the two receptors are translated into markedly different capacities for ligand accommodation.[2]

The indole–ethylamine core provides stable orthosteric recognition, whereas the C5-substituted structure probes subtype differences in the extended pocket. The volume, length, direction of extension, and flexibility of the substituent collectively determine whether these differences can be translated into effective receptor selectivity. Structural studies of 5HT₁B and 5HT₂B have therefore established a clear drug-design pathway: retain the essential orthosteric pharmacophore, extend a tunable substituent into a peripheral pocket with greater sequence and geometric divergence, and simultaneously evaluate its spatial compatibility with the target receptor and important off-target receptors.

8. Representative Related Chemicals and Pharmacological Probes

Table 1. Indole–Tryptamine Core Scaffolds, Endogenous Ligands, and Compounds for Studying Substituent Effects

Classification	CAS No.	Aladdin Product No.	Name	Specification or Purity	Product Features and Applications
Indole core	120-72-9	I104726	Indole	≥99%	Reference indole core scaffold for studying the electronic properties and substitution positions of the indole ring and the construction of tryptamine pharmacophores.
Parent tryptamine pharmacophore	61-54-1	T101154	Tryptamine	Moligand™, ≥98%	Parent indole–ethylamine pharmacophore for studying the contributions of the indole ring and basic ethylamine side chain to fundamental recognition by 5HT receptors.
Endogenous 5HT ligand	50-67-9	H303833	5-Hydroxytryptamine	Moligand™, ≥98% (HPLC)	Endogenous ligand of 5HT receptors, used as a reference in receptor-binding studies, functional assays, and evaluations of triptan receptor activity.
Endogenous 5HT salt-form reference	153-98-0	S111161	Serotonin hydrochloride	AR, Moligand™	Reference salt form of 5HT for receptor-functional assays, dose–response studies, and endogenous-ligand controls.
C5-methoxy tryptamine derivative	608-07-1	M124574	5-Methoxytryptamine	Moligand™, ≥97%	C5-methoxy-substituted tryptamine for comparing the effects of replacing the C5 hydroxyl group with a methoxy group on ligand polarity, electronic properties, and recognition by 5HT receptors.
α-Methylated tryptamine derivative	97469-12-0	M275267	α-Methyl-5-hydroxytryptamine maleate	≥99%	α-Methylated tryptamine receptor agonist for studying 5HT₂ receptor function and substituent effects on the ethylamine side chain.

Table 2. Triptans with C5 Substitution, Their Salt Forms, and Ergot Compounds Used in Structural Biology

Classification	CAS No.	Aladdin Product No.	Name	Specification or Purity	Product Features and Applications
Triptan with a C5 sulfonamide-containing substituent	103628-46-2	S333476	Sumatriptan	Moligand™, ≥98%	A 5HT₁B/₁D receptor agonist and classical triptan reference compound used to study C5 sulfonamide-containing substitution, receptor binding, and subtype selectivity.
Salt form of a triptan with a C5 sulfonamide-containing substituent	103628-48-4	S129681	Sumatriptan succinate	≥98% (HPLC)	Reference salt form of sumatriptan for 5HT₁B/₁D receptor-functional assays, dose–response studies, and comparisons of triptan activity.
Triptan with a C5 heterocyclic substituent	139264-17-8	Z129649	Zolmitriptan	Moligand™, ≥98% (HPLC)	A triptan bearing a heterocyclic substituent at C5, used to compare the effects of peripheral heterocyclic structures on 5HT₁B/₁D receptor activity and ligand conformation.
Triptan with a C5 nitrogen-containing heterocyclic substituent	144034-80-0	R613212	Rizatriptan	Moligand™, ≥98%	A triptan bearing a nitrogen-containing heterocyclic substituent at C5, used to study the relationship between C5 substitution patterns and 5HT₁B/₁D receptor function.
Salt form of a triptan with a C5 nitrogen-containing heterocyclic substituent	145202-66-0	R129638	Rizatriptan benzoate	≥98% (HPLC)	Reference salt form of rizatriptan for receptor pharmacology, quantitative analysis, and salt-form-related studies.
Salt form of a triptan with a C5 sulfonamide group	143388-64-1	N129630	Naratriptan hydrochloride	≥98%	Salt form of a 5HT₁B/₁D receptor agonist for triptan structure–activity relationship studies, receptor-functional assays, and pharmacological comparisons.
Salt form of a triptan with a C5 sulfonyl-containing substituent	181183-52-8	A129703	Almotriptan malate	≥98%	Salt form of a triptan bearing a C5 sulfonyl-containing substituent, used to study 5HT₁B/₁D receptor pharmacology and C5 substitution–activity relationships.
Triptan with an extended C5 sulfonyl-containing substituent	143322-58-1	E610098	Eletriptan	Moligand™	Contains an extended arylsulfonyl side chain at C5 and can be used to study the effects of peripheral-substituent volume, linker length, and conformational flexibility on accommodation by receptor subtypes.
Salt form of a triptan with an extended C5 sulfonyl-containing substituent	177834-92-3	E129729	Eletriptan hydrobromide	≥98%	Reference salt form of eletriptan for 5HT₁B/₁D receptor pharmacology, triptan activity studies, and salt-form comparisons.
Fused-ring triptan reference	158747-02-5	F610405	Frovatriptan	Moligand™, ≥97%	A tetrahydrocarbazole-type triptan reference for comparing the receptor-activity characteristics of a fused-ring scaffold with those of typical indole–ethylamine triptans.
Salt form of a fused-ring triptan	158930-17-7	F344297	Frovatriptan succinate hydrate	_	Salt form of frovatriptan for 5HT₁B/₁D receptor pharmacology and analytical comparisons of fused-ring triptans.
Experimental triptan/flexible C5 substitution	170912-52-4	D609948	Donitriptan	Moligand™	An experimental triptan with a relatively long, flexible substituent chain at C5, used to study how substituent length and conformational flexibility affect differences in activity at 5HT₁B and 5HT₂B.
Ergot ligand used in structural biology	511-12-6	D1074119	Dihydroergotamine	_	An ergot-derived serotonergic ligand and representative ligand used in 5HT₁B structural biology studies for comparing receptor-binding modes of ergot compounds and triptans.
Ergot salt form	6190-39-2	D137302	Dihydroergotamine mesylate	Moligand™, ≥98%	Salt form of dihydroergotamine for studies of 5HT₁B-related pharmacology, migraine receptor mechanisms, and comparisons with other ergot ligands.

Table 3. Pharmacological Probes for Studying 5HT₁B/5HT₂B Subtype Selectivity

Classification	CAS No.	Aladdin Product No.	Name	Specification or Purity	Product Features and Applications
Selective 5HT₂B agonist	160521-72-2	B1493349	BW-723C86 hydrochloride	Moligand™, 10 mM in DMSO	A 5HT₂B receptor agonist used in studies of receptor function and signal transduction and as a pharmacological comparator for the 5HT₁B and 5HT₂B subtypes.
Selective 5HT₁B inverse agonist	180083-23-2	S613446	SB 224289	Moligand™	A selective 5HT₁B receptor inverse agonist with receptor-antagonist activity, used to distinguish the binding and functional responses of 5HT₁B from those of other 5HT receptor subtypes.
5HT₁B/₁D receptor antagonist	148642-42-6	G167404	GR 127935 hydrochloride	≥98% (HPLC)	A 5HT₁B/₁D receptor antagonist used to verify the relationship between triptan effects and 5HT₁B/₁D receptor pathways.
Selective 5HT₂B antagonist	199864-86-3	R1430428	RS-127445 hydrochloride	≥99%	A high-affinity 5HT₂B receptor antagonist used for functional blockade of 5HT₂B, subtype discrimination, and agonist-response controls.

Note: The products listed above are representative Aladdin products relevant to scientific research. Their specific uses should be determined according to the product specifications, batch certificate of analysis (COA), and the intended reaction or evaluation system. Additional information on product specifications, grades, and COAs can be retrieved from the Aladdin website using the “product name/CAS number/product number.”

References

[1] Peroutka, S. J.; McCarthy, B. G. Sumatriptan (GR 43175) Interacts Selectively with 5-HT1B and 5-HT1D Binding Sites. European Journal of Pharmacology, 1989, 163(1), 133–136. DOI: 10.1016/0014-2999(89)90406-8.

[2] Wang, C.; Jiang, Y.; Ma, J.; et al. Structural Basis for Molecular Recognition at Serotonin Receptors. Science, 2013, 340(6132), 610–614. DOI: 10.1126/science.1232807.

[3] Wacker, D.; Wang, C.; Katritch, V.; et al. Structural Features for Functional Selectivity at Serotonin Receptors. Science, 2013, 340(6132), 615–619. DOI: 10.1126/science.1232808.

[4] Rodríguez, D.; Brea, J.; Loza, M. I.; Carlsson, J. Structure-Based Discovery of Selective Serotonin 5-HT1B Receptor Ligands. Structure, 2014, 22(8), 1140–1151. DOI: 10.1016/j.str.2014.05.017.

[5] Martí-Solano, M.; Sanz, F.; Pastor, M.; Selent, J. A Dynamic View of Molecular Switch Behavior at Serotonin Receptors: Implications for Functional Selectivity. PLOS ONE, 2014, 9(10), e109312. DOI: 10.1371/journal.pone.0109312.

For more related articles, see below:

D-DTTA Salts of Azaindole Chiral Amines: New Options for Chemical Splitting

The process path of EBL-3183: from indole to preclinical inhibitor

Base hydrolysis and activation experiments by diindolyl carbonyls

Bacterial indole test

From Indole to Azaindoles: A One-Nitrogen “Control Knob” for Tunable Properties and Scaffold Selection

Understanding Indoline: Analysis of the Structural Features, Properties, and Applications of the 2,3-Dihydroindole Scaffold

Indole Product Navigation: How N-Position State, the C3 Connection Handle, and Core Scaffold Variants Map to Synthetic Interfaces and Research Uses (Tables A–E)

Construction of 7-Azaindoles Using O-Vinylhydroxylamines: N-Oxide Activation and [3,3]-Sigmatropic Rearrangement

Synthesis of molecular block “two-sided” indole