GregoBase: status of site / codebase?
  • novusgordo
    Posts: 66
    As with many others, I've been making extensive use of GregoBase for preparing materials for my schola. The chant database is still being expanded and updated by many people. I'm growing increasingly concerned, however, at the state of the Web site itself.
    • The last commit to the GitHub repo was three years ago. In particular, the version of gregorio the site uses is badly out-of-date and seriously buggy.
    • M. Berten hasn't made any contributions to GitHub at all in the past year, which raises serious concerns about what happens if/when he goes the way of all flesh.
    • Finally, as the database expands, IMO there's a serious need to talk about conventions for things like how titles and other metadata are formatted. But it's hard to do that when the project lead isn't around!
    I know that on previous occasions, people on these forums have been able to help rescue sacred-music Web resources whose hosting or maintenance situations have become unsustainable. Given how important this particular site is to so many of us, at what point might we need to intervene?
    Thanked by 1Chrism
  • MatthewRoth
    Posts: 3,866
    Jacques Perriere also passed away. I have a lot of comments in the various comment fields and many thoughts (including that it is not a place to share new work, and vernacular work, not without better sorting) but it’s a lot of work. Also, I agree on titles, and no periods should ever be introduced into file titles either. But the titles are inconsistent and file names atrocious.

    It’s not about intervening. It’s about getting Olivier to respond.
  • novusgordo
    Posts: 66
    Re the better sorting:
    One thing I've been doing is prepending the "version" field of translations and other non-Latin scores with the two-letter language code: "EN:" for English, "VT:" for Vietnamese, and the like. But I'm also a little worried about doing that, inasmuch as I don't really have the authority to make decisions like that for the entire archive.
  • MatthewRoth
    Posts: 3,866
    It’s all reversible. We need a style guide. Some suggestions are also on the Gregorio mailing list. I don’t have time to do it this week but it is something which I will return to in due course.
    Thanked by 1novusgordo
  • igneusigneus
    Posts: 468
    Gregobase suffers the common fate of high-impact personal projects where the author never more can or wants to spend time on development and governance. It would be beneficial to the project if its development and governance was transferred to a larger body with a sufficient level of redundancy, clear community standards and processes to create, update and enforce them.

    (I virtually stopped contributing transcriptions to GregoBase once there were several years without publishing the database dump.)
    Thanked by 1novusgordo
  • MatthewRoth
    Posts: 3,866
    I agree.
  • So I just tried to go to Gregobase now, and it appears that the site is down, which is very concerning given the above conversation. I just get "There has been a critical error on this website. Learn more about troubleshooting WordPress."
  • ServiamScores
    Posts: 3,413
    Interestingly, the homepage works, and if you search directly for the first incipit letter it works. It's the "scores" page that's not working. But you can still get to things via:

    https://gregobase.selapa.net/incipit.php?letter=A (change the letter to whatever you need)
    Thanked by 1OMagnumMysterium
  • MatthewRoth
    Posts: 3,866
    I have no issues simply going to it via Google on my computer on my phone.
  • Works fine for me, except that the "logout" control isn't working properly. But all of this still makes me even more worried: just because the site is usable for now doesn't mean it will stay usable. (And while the codebase may be hosted on a git repository, AFAIK the database of chants only exists on M. Berten's server.)
  • igneusigneus
    Posts: 468
    Database dump is also committed to git. (Last updated three years ago.)
  • igneusigneus
    Posts: 468
    Also I remember @xmarteo mentioning somewhere that he has (or had at the time) direct access to the live database, so the bus factor regarding GregoBase data preservation may be at least 2.
  • MatthewRoth
    Posts: 3,866
    I don’t disagree, I’m just unable to troubleshoot if I can’t reproduce the issue.
  • xmarteo
    Posts: 13
    Yesterday's outage might or might not be related to https://en.wikipedia.org/wiki/PHP2Shell

    I think Olivier did update PHP because the PhpMyAdmin login page has been updated (though not PhpMyAdmin itself).

    If he did not update PHP however, it is only a matter of time before someone brings Gregobase down for good.

    As implied above, I do have access to Gregobase's PhpMyAdmin, so I can dump the DB and make manual edits into the DB, including running SQL commands. Not having access to the server itself I cannot easily cron a DB dump, which would be, if not ideal, at least a first-step contingency measure.

    Here is an emergency dump: https://ftp.marteo.fr/grego.sql

    I fully agree with @igneus that "it would be beneficial to the project if its development and governance was transferred to a larger body with a sufficient level of redundancy, clear community standards and processes to create, update and enforce them" and I guess Olivier would be fine with it - but which body? CMAA, Una Voce?

    Running Gregobase doesn't cost much but the ~300€/yr has to come from somewhere, and more importantly, Gregobase has become very technologically outdated (using manual string comparison to store diffs in-DB as a means of version control has not been done since the 90s!) and needs further development efforts.

    Edit: regarding the developmental "death" of Gregobase: just because nothing is committed to Github does not mean nothing is done, because Olivier patches prod directly and does not always commit his changes :p
    Thanked by 3probe tomjaw novusgordo
  • xmarteo
    Posts: 13
    Finally, I proposed a number of editorial guidelines here a couple years ago: https://groups.google.com/g/gregorio-users/c/3P1hHtW1XOI/m/_sR4D4lgAQAJ

    It was not agreed upon, but some contributors made an effort to stick to them.
    Thanked by 1tomjaw
  • xmarteo
    Posts: 13
    I just ran a couple audit queries that I wrote back in 2021 to measure the quality of the data in Gregobase and it is not great. Hundreds of misattributed sources, misattributed versions, junk scores (almost empty GABC or random "tests" by people). I'll do some emergency cleanup today as I have a couple hours on my hands.

    https://grego-analytics.marteo.fr/ gives some insight (not updated live, so if you check this link after I did my cleanup you will have a very outdated picture).

    Also reminder that emptying Gregobase entries is /not/ the way to remove duplicates. There is an auto-redirect feature ("duplicateof" field in the chants table).
    Thanked by 2probe tomjaw
  • MatthewRoth
    Posts: 3,866
    I would be happy to start following these guidelines more, frankly.i have not had a lot of time but since Jacques Perriere passed I coincidentally have had less need to get scores.

    (I also don’t really see why in the absence of something better we don’t just come to a community consensus on at least some of these issues.)

    I would propose the CMAA myself; I also note that it is beyond time to go to the latest version of Gregorio…
    Thanked by 1tomjaw
  • xmarteo
    Posts: 13
    CMAA has the drawback of not being an international association; whereas Gregobase was created by a Belgian to demonstrate the capabilities of a system built by a French and maintained by a Brasilian, an American an a Czech, and most of its contributions came from an Australian.

    Given the importance of Gregobase in the ecosystem, it could be an ad-hoc body as well, if we can somehow find funding.
  • MatthewRoth
    Posts: 3,866
    That’s true, although in practice it is international.
  • probe
    Posts: 270
    Thank you @xmarteo I was hoping to avoid a htttrack dump ;)
  • ian_udell
    Posts: 34
    Maybe a bad suggestion, but possibly Source & Summit could be the governance that could take it over? They already have their score editor, and I'm sure it would be very useful to anyone who has the digital platform to be able to look up any chant and it be there.
  • xmarteo
    Posts: 13
    They are a for-profit company, instinctively I am wary of the idea.
  • There is a real anxiety right now in the FOSS community about companies taking over FOSS projects, dual-licensing them, and then gradually killing the FOSS version. I'm definitely not accusing Mr. Bartlett of having ulterior motives, but I suspect that many users of gregorio would be ill at ease with his company taking over governance of a resource like GregoBase.

    There's also the matter of licensing non-public-domain liturgical texts such as the ICEL translations: it's often much easier to do so for not-for-profit purposes. IMO this should also be a consideration.
    Thanked by 1xmarteo
  • xmarteo
    Posts: 13
    So, the good news is, Olivier is alive, and the outage was, in fact, due to him updating PHP following the the PHP2Shell vulnerabilities, so Gregobase isn't going anywhere just yet.

    What I suggest for the present is that, short-term, someone should step up as lead maintainer, receive admin rights on the Github repository, fix the legitimate bugs that pile up in the Github issues, and arbitrate on the pending pull requests. The right candidate is obviously @igneus but does he want to?

    In parallel, I would advocate the setting up of a test instance with a modern stack (latest Debian, PHP, Wordpress and Gregorio) to a) test added features outside of prod (currently the only instance afaik) and b) fix the issues that might arise due to such a jump in time. I can set that up if needed on a marteo.fr subdomain.

    Finally, longer term, Gregobase could move out of selapa.net to its own server thanks to the generous funding of donor(s) that have already and will in the future manifest themselves. This would allow low-level maintainance of the prod instance to stop relying on a single person. It is also at that point that, since the running of the site would no longer be only under Olivier's responsibility, some sort of board should be put in place to define and enforce community guidelines.