It's possible for some hymns, but there are difficulties with others. For example, the only way to notate a dotted rhythm is with a tristropha followed by a punctum, but that also implies a repercussion on the third note of the tristropha, and possibly the second note as well, depending on which school of chant interpretation one follows. Rests are also problematic, because the normal way to notate them is with a full bar line, which also requires a ritardando, or a half bar line, which requires diminution of the preceding note. Otherwise you're getting into nonstandard styles of Gregorian notation. Try copying and pasting this here:

centering-scheme: english;
%%
(f3)Ho(e.)ly(e) God,(ed) we(e) praise(fg) Thy(f) Name;(eee) (;) Lord(g.) of(g) all,(gf) we(e) bow(ih) be(g)fore(g) Thee.(f.) (;)

You can keep going with it, but suppose you want to notate the eighth notes that are customarily sung in the last phrase in many places. You could use a pes initio debilis, but that suggests a note of more diminished (half) value followed by one of diminished (short) value, not two of half value. There's really no way to notate that, and you can't double all the note values because of the repercussions that would imply.

@madorganist

Thanks for info. I just set this one, which kinda works, but I see that once you get into something that is not a plainchant style hymn, you got problems.

You could dot everything instead

centering-scheme: english;
%%
(f3)Ho(e.e.)ly(e.) God,(e.d.) we(e.) praise(f.g.) Thy(f.) Name;(e.e.e.) (;) Lord(g.g.) of(g.) all,(g.f.) we(e.) bow(i.h.) be(g.)fore(g.) Thee.(f.f.) (;)

but then you run into the repercussion problem: Ho-holy God, we praise Thy Na-ha-hame; Lo-hord of all, we bow before Thee-hee.

I think I’d only attempt this with either proper chant tunes (adoro te devote but with English text) or something composed in a deliberate chant style. I have an example on my work computer I’ll try to share tomorrow.

It seems to me you're making this harder than it is: the inventors of modal and mensural notation had no problem ditching the baggage of a different style of singing and hijacking extant glyphs:

(f2) 3In(gv)fi(h)(,)nite(ixi/h) thy(g) (,) vast(h.J) do(IK)(,)main;(jv.) (,) Ev(kv)er(k)(,)last(ixj/i)ing(h)(,) is(ixi/h) thy(g)(,) reign.(fv.) (::)

ServiamScores,

That is a beautiful melody. Have you written an organ accompaniment for it?

Here's an organ accompaniment to my hymn above.

ServiamScores,

I took a little time to look through the scores and have a question. Why in the organ accompaniment is there an F natural on the second note? If do is an A, then wouldn't la be F#?

this fixes it I think

Years ago, I wrote a Mass setting based on Shaker chant. I originally used Gregorian notation for it, because I Thought it was the best way to represent the free-but-not-quite-free rhythm I was going for. Several people pointed out that it felt like a weird, out of place, affectation.

On the other hand, I did music for an Eastern Rite Anglican (I don't want to talk about how weird that is, please) monastic community, and I set all the Byzantine stuff using gregorian notation and it worked really well for that.

There are two inter-related problems with using a non-standard notation system:
- Can it communicate what it needs to?
- Will the people using it accept it?

As anyone who has set Gregorian chant to five line round notes knows, the second is actually more important than the first.

Maybe you'd have done better with individual words and a dictionary, but what I meant was that, in my example as well as historically, virga can be used for longa, punctum for breve &c.