There's a lot to unpack here.
First, the notion of "original keys". Anything originally written in "high clefs"/ chiavette (basically treble/mezzo/alto/tenor or baritone, or some variation thereof) would have been transposed down a third or 4th for the male choruses of the time. Why would they do that? For the same reason we notate Mode VII chant from g to g....it's the place where the whole and half steps fall in the proper order for the mode. Most of us wouldn't pitch Mode VII chant "at notated pitch" but a 4th (or for Mode 5, a 3rd) lower, And the "major key sounding modes" became more popular during the Counterreformation, and thus so did high clefs. The reason this matters is that those same high clef pieces are the ones that can be most successfully done by an SATB choir (which is anachronistic to this music). But even then, the soprano and tenor parts can be uncomfortably high and benefit from a drop of a second, even though the altos will not love you for that. So, first, find an edition in a pitch that works for your choir. And if the only available edition needs transposition for your group, make a new transposed edition rather than expecting them to sing a g or f when their throats are set for the Bb they see on the page.

Second, find an edition that respects notational conventions that your choir expects. Editions with a lot of editorial nonsense (detailed dynamics, articulations) should be avoided. I personally hate Mensurstriche, because choirs expect real barlines and the same number of beats in every measure. Musica ficta must be identifiable as editorial. An edition without any ficta is preferable to one in which all accidentals are displayed as Holy Writ. Ligature brackets? Important if changing underlay, but easy for a choir to ignore. I don't like to add melisma slurs in this repertoire because I find they clutter the page. And editions which attempt to represent original notation at the expense of those conventions (*cough* Renata Calcaterra) are totally useless to me.

One place I'd disagree with you is note values. I generally use original note values in 16th century music, even in tripla sections. My singers have no issue with whole notes getting the beat, and they are amateur non-early-music people.

An edition should contain everything that is expected to be sung, including incipits and alternatim chant. My solutions to this problem are geared to a church choir and are not historically informed. I will use Mass VIII as the incipit to any Gloria I can get away with, because it's what priests are most comfortable with. And if I have added chant, it's Solesmes, because it's easy and that's what they're used to. If I go to great effort and find exactly the right version of a chant that they know, then they will stumble over the differences. In any case, anything added should be sourced so people know. And chant must be in square notes, not round notes with quasi-ligature slurs. If the program won't do it, use Gregorio or some other program that will, and paste it into a pdf.

I regularize spelling of text to the LU, and try to hyphenate correctly. But sloppy text isn't a big dealbreaker, Ugly cramped notation is.

So when does one do a new edition? A big determiner is whether an xml file is available. I'm just not going to re-enter a piece, especially a big one like a Mass. If a piece MUST be re-edited, and it can't be done easily, well there's a lot of music on the music tree...NEXT! I'd rather input music that hasn't been edited. But when?
1. Pitch off more than M2.
2. Combination of other little things.
I can live with mensurstriche or altus parts in octave treble, even though I know they will cost rehearsal time. But at some point the costs add up to where I feel I have to re-edit. A thing that's sub optimum for ME (wrong incipit, when one can just play the right one) is not the same as one that's sub optimum for THEM.

Ooh! That’s a great answer. Thank you!

Ah, yes; on the incipits, our organist isn’t the DM, so we have to tell the organist what to play and the key if it’s not the same as the one usually used. So it adds an extra step. It is not the end of the world, and you’re right; the director’s problems versus the choir’s don’t overlap.

One place I'd disagree with you is note values. I generally use original note values in 16th century music, even in tripla sections. My singers have no issue with whole notes getting the beat, and they are amateur non-early-music people.

Fair, and I did say “probably”, at least with respect to our case. :)

Also, there’s probably somewhere else online that explains it, but:

What’s up with the notes on the original clef and then two notes at the beginning of the actual staff? I have attached the beginning of the NEV for the O quam gloriosum. I can understand original clef and first starting note, but there’s something else going on. And many editions omit the two notes on the actual staff…

Why is text sometimes italicized? Is it because the original isn’t clearly laid out?

And is there a book that I should read or something? Because at the end of the day, maybe some of this stuff doesn’t matter, but it’s there, so I’m bound to want to know about it (and someone in my choir is going to ask me, so not having an answer frustrates other people too…)

I think that most Palestrina masses are on cpdl.org in MIDI or even MusicXML so I would import it to Musescore, transpose in three clicks if needed, and press } enough times to widen the measures if the text is squashed up.
(Off topic, but similarly for Gregorian chant I download the gabc from Gregobase, make a few changes like sprinkling (z) then use run.gregoriochant.org to save as PDF for printing, so that my choir has a nicely spaced score that shows clearly what syllables belong to each note.)

The notes at the left of the stave are the incipit, ie the beginning, to show scholars how it was originally notated. As CHGiffen said, the other two inside are the ambitus, the vocal range.

There is a huge amount of scholarly writing on Palestrina. If you have library access to eg JSTOR spend some time browsing and see if any titles appeal to you. Like Composers at Work, The Craft of Musical Composition, chapter 11, Jesse Ann Owens. Or simply listen to recordings and see whose style you like.

"But I don’t understand the conventions. Sometimes it’s one note on the original clef, sometimes not…" Do you mean sometimes the incipit shows one note, sometimes more? I think it's just whatever the transcriber thinks is useful, see:

https://www.helenhooker.co.uk/blog/2021/9/28/cracking-the-code-6pwhx

https://stdionysius.lochac.sca.org/collegeprojects/twmnlt.pdf
says:
It is commonplace in most transcriptions of period music to give an “incipit” of the original as your starting
bar. You can do this by:
• Giving the original clef, key and time signature
• Noting the first few notes of the piece, especially if the notes are common to all (or most) parts
• Adding the common material to the bar, along with any rests the part gives before its music starts
• Giving an indication of note value reduction, if you want to use it
• Giving a double barline, then commencing your transcription

If you really want to dive deeply into this topic, I'd suggest reading through (or at least glancing over) the style guide for A-R Editions, who publishes the journal Recent Researches in Music. Their editions are always very clear visually; information about music typesetting begins on PDF page 14 (numbered as p. 10).

Coming to italicized text... From my understanding this appears when the original only states "ij" for a text repetition (you know that from the Kyriale) and the editor has to lay out the text by himself.

Btw I disagree with some of your syllabifications, but in the end that's something your choir can easily correct with a pencil in their hand (which they should always have). Some scholars suggested that the syllabification is one way to determine whether a composer had German or Italian pronounciation in mind (mag-nam vs ma -gnam is a good example) - ok, that's no question for Palestrina.

The ambitus indication is one of the most important editorial features for me as it's not always easy to find early music which fits a modern SATB choir (as we discussed).

I think a good way to learn all that is to have at least a glimpse on some original prints and maybe even transcribe something.

When do I make a new edition? It depends on how easily I can do it. I use lilypond, so when I have a source file, it's easy. Some cpdl Lilypond editors are doing a fantastic job, much more professional than I could, some are really good in reading early music but not so good in presenting it in a convenient way. Pitch: a major second, sometimes even a minor third is ok (but I work with more experienced singers). When there is too much to correct I will make a new one.

And: When you prefer editions notated in quarters instead of half notes - victoria.uma.es has it so. Although I prefer half notes, I often come back to this great resource.

Right, but I mean the actual words are given in italics. I ask because in an edition of the Sicut Cervus which I looked at, one of the “ita desiderat” in the tenor has a different distribution over the notes than I’m used to seeing without explanation beyond the italics. It hadn’t occurred to me that this part could be subject to an editor’s decision; at least the editor of the first edition of the Missa ACM which I examined this week explained that the Kyrie in Palestrina’s masses is not laid out in full and so singers (and now editors) have to figure it out.

smt, I know that some of my choices for syllabification, are the older style but that’s what we see 90% of the time, etymologically the prepositions and adverbs staying together makes sense (I sometimes wonder if editors don’t recognize this…), and then as far as the consonant clusters go, it has a long history; the editor of the Latin packages for LaTeX doesn’t want to allow for -ct, but it has a history dating to late antiquity per his own remarks to me on the subject. (This is annoying insofar as if you’re going to create hyphenation rules that alone for almost everything that I want, do that too, and make it easy for me to make other changes like p-t to -pt).

Yeah I like the victoria.uma.es editions too. I would like to try more with half notes getting the beat or even whole notes since Jeffrey has convinced me to give it a shot.

I go back and forth on italics. In later 16th-century music, "ij" tends to fill words that run one note per syllable, so there's only one way to do it. I tend to reserve them for things that are legitimately added: missing words, or alternative readings where the manuscript text is slightly different from the modern liturgical text. If there's a "prime directive" in music editing, it's "show your work".

I will usually do incipits. Original clefs and mensuration are important; notes notsomuch. If I'm working from a secondary source, I generally don't, because I generally don't have that info, and if it isn't a "scholarly" edition, why confuse the issue with the scholarly apparatus? Besides, doing them well is always a bit of a PITA, since not even Dorico allows for use of all the SMuFL glyphs as easily as the standard set.

At the 2021 CMAA virtual colloquium, David Hughes did a great presentation on editing early music https://connect.churchmusicassociation.org/page/2021vc (members only). John Caldwell's book, Editing Early Music (Oxford UK: Clarendon Press, 1985) has some useful ideas as well.

We read the same score and change beginning pitch often. You really dont need to have it written in the actual key you are performing in unless that is a problem for learning and rehearsing.

As for specific keys, I think the idea of religiously sticking to one or the other is complete hogwash. Pitch the motet where it makes the most sense for the singers you have. As long as the vocal ranges sit properly for each part, you can take it up or down as needed. What good is singing something in the “original” key if it sounds like trash because it’s too high or too low?

Sure. But like with the propers, we can sometimes hit the note, it’s just…well, should we…and my guys used to hate high notes so we never drilled. But now in warm-ups I sometimes go an octave above our usual top notes and descend two octaves, well below the lowest La of any mode 2 (or occasionally 1) chant.